Hello,
in order to run simulations/optimizations faster I'm trying to figure out which hardware set-up has the most impact on performance. I'm looking at the AMD CPU FX lineup and it seems that an 8 core would be preferable since I could be running 8 optimization iterations at a time.
Is there a guide or recommendations on selecting hardware in order to have optimizations run as fast as possible ?
The faster the CPU the better. You can compare different CPUs online.
Having multiple cores is more useful when performing trading system optimizations and AI optimization using the PBIL algorithm (AI -> Optimizer in menu)
I did some testing on an I7 4790 with 8GBs of RAM and what I found using an PBIL optimization for a specific system, was that CPU load was around 35% (all 8 threads were being used) and memory load was around 3GBs. In trying to figure out what would be a great setup for a dedicated box for QS, I would like to know what would make more sense in terms of where to concentrate spending efforts. If the CPU is not being utilized at a high load, where is the bottleneck ? Hard disk ? Memory ?
Is it pretty much the same to have an I7 4790, an I7 4790K, an I7 6700K but an SSD drive makes a difference ? Faster memory ? Any Windows registry settings that need to be changed to improve performance ?
I guess you made sure the "Enable Multithreading" option is selected when creating your PBIL item.
Is the following setting checked (Accounts -> Application Settings -> Memory Management -> Load quotes in memory...) ?
Yes, "Enable Multithreading" is always left on both at the AI optimization and as part of the "More Settings" of the simulation. Regarding the "Load quotes in memory", I had it unchecked. I remember checking this option and unchecking it after some problems in the past( the machine I'm running is a Virtualbox with almost 5GB of Ram allocated and an AMD Athlon X4 630 CPU). I did check it again to see what happens and CPU usage does go up and speed seems to be much faster. For the time being no problems and this seems to help a lot. So thanks for reminding me of this option !
As for the hardware, any tips on where to concentrate spending efforts ? What is the sequence ? CPU with more cores, then CPU speed, faster hard drive, faster memory ? Have you done some benchmarking with a recommended setup? Is it detrimental to run an PBIL along to with an GA at the same time ? The ideal would be have the computer at a very high CPU load during optimization runs, how is the best way to achieve this ?
I would say the most important by priority:
CPU Speed
CPU Cores
Hard drive
Faster memory
You can run a PBIL and GA at the same time as long as you have enough cores to do this but the GA algorithm does not use multithreading.
Usually when you run a PBIL optimization, the algorithm will use the following number of cores (Number of cores - 1), so you are always left with one core. If you run the GA at the same time, the GA algorithm will use that remaining core.
I am running QS on an i7-3770 processor with 32 GB memory under Windows 7. QS runs on an SSD and quotes are in memory. Despite the fact that a 4th or 5th generation processor would perform better than my 3rd gen. does, I am facing the same situation: CPU usage is rarely above 50% in average.
While I agree with what was written above, my experience is, once you already have a rather sophisticated system, you will have two bottlenecks:
a) the PBIL algorithm itself
b) communication between memory controller and memory
a) sounds strange, but for a single application parallel computing in general and as performed in multithreading PBIL is much about waiting for the CPU. Example: if the simulation run of one population member in a generation takes longer that the others, the CPU will be under-utilized as it has to wait for the completion of the generation.
b) 4 or 8 cores are permanently requesting quotes information from memory through memory controller and usually through only 2 memory channels, so CPU needs to wait for a free memory channel. CPU cache is normally too small for the amount of data we are dealing with, so caching withiin the CPU won't help.
There is nothing you can do about a). Regarding b) I might think about Quad-channel for my next system, but haven't made any plans yet.
Thanks to both for your answers, I found them very interesting and they left me thinking and doing further research. And also thanks to QS for devoting time to improving performance under the hood.
So, a Quad channel controller for the memory and dividing the memory in as many sticks as possible would be one of the ways to go it seems. I was thinking of 32GB divided in 4 sticks. I don't know SystemTrad.de, if from your experience, this is enough in terms of memory or you would be looking at a setup with more than 8GB per channel. I'm also curious since there are some options out there that handle quad channels, even cheap old Opterons, what you are potentially looking at.
Since this isn't an application where we just throw in the lastest GPU and CPU and expect better frames per second and a 100% load, I find this discussion extremely useful both to better understand how it works, and also as a money saver.
I would disagree with "dividing the memory in as many sticks as possible" - ideally each memory bar should have its own controller and own channel, so on a Quad-Channel system 4 x 8 GB memory would be perfect.
32 GB is more than I ever needed and no, I am not looking at cheap old Opterons, check cpubenchmark.net for instance, the cheap ones are relatively poor performance-wise
That's what I was thinking, under the assumption that there are no more than four memory banks per CPU, so it is better to have a 4x8 than 2x16 with a quad memory controller CPU. So thanks for pointing me in this direction, I had totally missed the higher than dual channel feature.
Trading financial instruments, including foreign exchange on margin, carries a high level of risk and is not suitable for all investors. The high degree of leverage can work against you as well as for you. Before deciding to invest in financial instruments or foreign exchange you should carefully consider your investment objectives, level of experience, and risk appetite. The possibility exists that you could sustain a loss of some or all of your initial investment and therefore you should not invest money that you cannot afford to lose. You should be aware of all the risks associated with trading and seek advice from an independent financial advisor if you have any doubts.
0
