New servers inferior CPU performance in Help

edited June 2017 in Help
Using a 5€ VPS (Frankfurt) two months now, I wanted a second 10€ one.

To my surprise, the performance of the new vps was much lower using exactly the same OS (FreeBSD 11) and settings. Recompiling the same project needed almost 30% more time; same when building other FreeBSD ports.

At first I assumed it was something random and destroying that instance I created a third and a fourth and a fifth one on different data centers (Amsterdam, Frankfurt) but the problem persists!

According to system info, CPU is different too!
Old VPS is 2400Mhz ID=0x306c1 model=0x3c but in all new ones is 2394Mhz ID=0x306d2 model=0x3d

After further testing with other OS distros and benchtests I came to the conclusion that Vultr decided to use lower-performance processors in their new boxes.

Any idea what's going on with this really odd situation?
I do like Vultr, but performance is supposed to increase over time not to decline!
Tagged:

Comments

  • @Athan I'm running FreeBSD on 5 different instances. What benchmarks did you use, and what results do you get? I'll post mine, see what's going on. - You shouldn't be seeing any major speed difference at all.

    Did you install from the FreeBSD custom install iso, or use the vultr automated FreeBSD option?
  • edited June 2017
    I tried both FreeBSD ISO as well as Vultr's image with no noticeable difference.

    I've run many tests including sysbench, unixbench, though I don't recall the exact numbers now (I'm at work)

    However I remember the most representative real world test, port building.

    Perl5.26 port timed builds
    Old (cpu id=0x306c1): 5:01
    New (cpu id=0x306d2): 6:53 (38% slower)
  • edited June 2017
    From what I can gather, "0306c1" is 4th generation "Haswell" and "0306d2" is 5th generation "Broadwell". For the same CPU speeds etc. if anything the new CPU should be faster.

    I've not had a chance to speedtest yet, but every CPU of mine is different! (Note the only 2 with the same speed and family/model/stepping have a different CPU string (which may or may not mean anything))

    I have the following,

    16:55 (28) "/tmp" root@lapcat# do-all-hosts "grep -Ei 'extended' /var/run/dmesg.boot"
    ___________________________________________________________________
    jamie@catflap: 'grep -Ei '^CPU:|origin=' /var/run/dmesg.boot'
    catflap: CPU: Virtual CPU 714389bda930 (3392.26-MHz K8-class CPU)
    catflap: Origin="GenuineIntel" Id=0x306c1 Family=0x6 Model=0x3c Stepping=1
    ___________________________________________________________________
    jamie@catnip: 'grep -Ei '^CPU:|origin=' /var/run/dmesg.boot'
    catnip: CPU: Virtual CPU e7da7129d3ee (2400.08-MHz K8-class CPU)
    catnip: Origin="GenuineIntel" Id=0x306c1 Family=0x6 Model=0x3c Stepping=1
    ___________________________________________________________________
    jamie@catseye: 'grep -Ei '^CPU:|origin=' /var/run/dmesg.boot'
    catseye: CPU: Virtual CPU 714389bda930 (3600.16-MHz K8-class CPU)
    catseye: Origin="GenuineIntel" Id=0x306c1 Family=0x6 Model=0x3c Stepping=1
    ___________________________________________________________________
    jamie@catmint: 'grep -Ei '^CPU:|origin=' /var/run/dmesg.boot'
    catmint: CPU: Virtual CPU 714389bda930 (2400.11-MHz K8-class CPU)
    catmint: Origin="GenuineIntel" Id=0x306c1 Family=0x6 Model=0x3c Stepping=1
    ___________________________________________________________________
    jamie@cattail: 'grep -Ei '^CPU:|origin=' /var/run/dmesg.boot'
    cattail: CPU: Virtual CPU a7769a6388d5 (2394.51-MHz K8-class CPU)
    cattail: Origin="GenuineIntel" Id=0x306d2 Family=0x6 Model=0x3d Stepping=2

    I guess I could run some tests on cattail and catnip, as they are both still on FreeBSD 10, whilst catmint is on 11....
  • I really don't know if vCPU ID means a lot but in real world tests new instances are 35-50% slower even though both old and new processors are clocked @ 2.4Ghz

    And a 40% performance loss is something really serious imho. I understand that Vultr might be trying to squeeze the cost but still...
  • edited June 2017
    @Athan I agree, it's one hell of a hit.

    I'm puzzled, though, that the features and speed etc. of the reported CPU are basically the same - is it possible that this is due to some different resource setting on new KVM guests rather than the listed CPU itself?

    I'm about to clone one of my original instances and compare the two, though at this stage I guess I'm only going to confirm your findings...

    Cheers
  • edited June 2017
    Maybe you're right Jamie regarding the KVM setup because every testing instance I've created across Vultr's network looks equally slower than the old ones.
    Not a very pleasant finding unfortunately...
  • edited June 2017
    @Athan - Well, my results are quite surprising.

    I have an instance in Sydney running FreeBSD 11. (catmint)

    I snapshotted it, and used it to create 4 new $5.00 instances, in London, Paris, Seattle, and Sydney.

    I ran the build of perl on each machine. slept 30 minutes, then repeated.

    As with you, I got some results around 5 minutes per server, and one around 7.

    So in summary:

    It's not effecting all new instances, fortunately.
    It's also not related to the CPU (you'll see I got 3 new cpus and 1 older one on my new instances)

    The only thing I've noticed that tallies with the slower operation is the slower one reports 2.394 Ghz, and the others 2.4Ghz - obviously that difference in itself isn't the issue, but it's the only thing I can see that is different. You may be able to see something here that I've missed!

    All have the same CPU features:

    Features=0x783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2>
    Features2=0xfffa3203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
    AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
    AMD Features2=0x21<LAHF,ABM>
    Structured Extended Features=0x728<BMI1,AVX2,BMI2,ERMS,INVPCID>
    XSAVE Features=0x1<XSAVEOPT>

    My "real" instance (Sydney):

    CPU: Virtual CPU 714389bda930 (2400.11-MHz K8-class CPU)
    Origin="GenuineIntel" Id=0x306c1 Family=0x6 Model=0x3c Stepping=1
    Wed 21 Jun 2017 21:00:01 BST 298.03 real 256.35 user 36.26 sys
    Wed 21 Jun 2017 21:35:00 BST 299.62 real 257.38 user 37.00 sys
    Wed 21 Jun 2017 22:10:00 BST 299.43 real 257.99 user 35.99 sys
    Wed 21 Jun 2017 22:45:00 BST 302.04 real 259.46 user 37.14 sys
    Wed 21 Jun 2017 23:20:03 BST 301.31 real 258.40 user 37.43 sys
    Wed 21 Jun 2017 23:55:05 BST 302.38 real 258.28 user 38.50 sys
    Thu 22 Jun 2017 00:30:08 BST 299.90 real 257.69 user 36.88 sys
    Thu 22 Jun 2017 01:05:09 BST 303.30 real 260.30 user 37.58 sys
    Thu 22 Jun 2017 01:40:13 BST 302.68 real 259.72 user 37.46 sys
    __________________________________________________________________________________________

    London:

    CPU: Virtual CPU 714389bda930 (2400.07-MHz K8-class CPU)
    Origin="GenuineIntel" Id=0x306c1 Family=0x6 Model=0x3c Stepping=1
    Wed 21 Jun 2017 21:02:04 BST 302.51 real 259.34 user 37.86 sys
    Wed 21 Jun 2017 21:37:07 BST 299.91 real 257.48 user 37.25 sys
    Wed 21 Jun 2017 22:12:08 BST 307.46 real 263.45 user 38.45 sys
    Wed 21 Jun 2017 22:47:16 BST 301.19 real 258.95 user 37.08 sys
    Wed 21 Jun 2017 23:22:18 BST 301.53 real 258.35 user 37.73 sys
    Wed 21 Jun 2017 23:57:20 BST 301.61 real 259.55 user 36.85 sys
    Thu 22 Jun 2017 00:32:23 BST 304.12 real 260.83 user 38.07 sys
    Thu 22 Jun 2017 01:07:28 BST 309.07 real 264.68 user 39.13 sys
    __________________________________________________________________________________________

    Paris:

    CPU: Virtual CPU a7769a6388d5 (2400.11-MHz K8-class CPU)
    Origin="GenuineIntel" Id=0x306d2 Family=0x6 Model=0x3d Stepping=2
    Wed 21 Jun 2017 21:04:36 BST 328.89 real 281.25 user 42.04 sys
    Wed 21 Jun 2017 21:40:06 BST 303.13 real 258.30 user 39.47 sys
    Wed 21 Jun 2017 22:15:10 BST 317.32 real 269.51 user 41.84 sys
    Wed 21 Jun 2017 22:50:28 BST 316.12 real 270.35 user 40.28 sys
    Wed 21 Jun 2017 23:25:45 BST 314.15 real 267.76 user 40.78 sys
    Thu 22 Jun 2017 00:01:00 BST 312.26 real 266.37 user 40.49 sys
    Thu 22 Jun 2017 00:36:13 BST 310.29 real 264.05 user 40.77 sys
    Thu 22 Jun 2017 01:11:24 BST 309.83 real 263.99 user 40.19 sys
    __________________________________________________________________________________________

    Seattle:

    CPU: Virtual CPU a7769a6388d5 (2400.10-MHz K8-class CPU)
    Origin="GenuineIntel" Id=0x306d2 Family=0x6 Model=0x3d Stepping=2
    Wed 21 Jun 2017 21:01:13 BST 296.13 real 252.10 user 38.53 sys
    Wed 21 Jun 2017 21:36:10 BST 294.93 real 252.04 user 37.53 sys
    Wed 21 Jun 2017 22:11:06 BST 296.86 real 252.30 user 39.08 sys
    Wed 21 Jun 2017 22:46:03 BST 293.95 real 249.89 user 38.71 sys
    Wed 21 Jun 2017 23:20:58 BST 292.90 real 250.03 user 37.58 sys
    Wed 21 Jun 2017 23:55:52 BST 294.05 real 249.91 user 38.59 sys
    Thu 22 Jun 2017 00:30:47 BST 289.68 real 246.16 user 38.15 sys
    Thu 22 Jun 2017 01:05:37 BST 292.31 real 248.42 user 37.10 sys
    __________________________________________________________________________________________

    Sydney:

    CPU: Virtual CPU a7769a6388d5 (2394.50-MHz K8-class CPU)
    Origin="GenuineIntel" Id=0x306d2 Family=0x6 Model=0x3d Stepping=2
    Wed 21 Jun 2017 21:04:55 BST 390.95 real 333.06 user 51.17 sys
    Wed 21 Jun 2017 21:41:27 BST 422.11 real 360.40 user 55.05 sys
    Wed 21 Jun 2017 22:18:30 BST 437.31 real 374.04 user 55.95 sys
    Wed 21 Jun 2017 22:55:49 BST 423.73 real 360.89 user 56.01 sys
    Wed 21 Jun 2017 23:32:53 BST 428.76 real 366.19 user 55.59 sys
    Thu 22 Jun 2017 00:10:03 BST 408.01 real 343.91 user 53.13 sys
    Thu 22 Jun 2017 00:46:52 BST 405.81 real 345.26 user 53.63 sys
    Thu 22 Jun 2017 01:23:40 BST 407.90 real 348.58 user 52.48 sys
  • edited June 2017
    I just created an instance in Frankfurt, and one in Amsterdam - both from the vultr freebsd 11 install option, and timed the perl5.26 install.

    The one in Frankfurt - 2400Mhz - time approx 300 seconds.
    The one in Amsterdam - 2394.50Mhz - time approx 440 seconds.

    So again, one with the same "slow" time that is 2394mhz, and one with the same "fast" time, which is 2400mhz.

    So it does seem to be the link somehow. It would be interesting if anyone else with a different OS could compare a 2394 instance against a 2400 one.
  • edited June 2017
    Yesterday I've created two instances in Frankfurt; both 2400Mhz and fast.
    I suspect Vultr people read this forum, because all those created last week were 2394 and slow. ;)

    In any case I'm grateful because 40% performance loss was a showstopper issue, at least for me!
  • edited June 2017
    Glad you managed to get them! Something to look out for in the future, anyway!

    I'm clinging onto this one as long as I can :smile: catseye: CPU: Virtual CPU 714389bda930 (3600.16-MHz K8-class CPU)
  • edited June 2017
    You're a lucky guy... :)
    Back then when 3600 was available here, I was still fighting with colo dedicated, great performance and resources but costly for my modest needs...
  • I've had a host online since 1999 - back then they were dedicated, and indeed, a hell of a lot more expensive than they are now. And those hosts ($60+) were less powerful than a $5 vultr instance today.

    kids today, they don't know how lucky they are etc.!
  • They really don't! :wink:
  • edited June 2017
    If the new 2.4Ghz cpus are Broadwell-EP based E5-26xx v4, then depending on application/usage could be slower as you may have found the same thing discussed at https://www.webhostingtalk.com/showthread.php?t=1624698 that E5 26xx v4 L3 latency is almost double that of E5 26xx v1 with more L3 cache size though and that to better leverage the performance of E5-26xx v4, you would need to custom recompiling the application to better leverage newer cpus

    * https://www.webhostingtalk.com/showthread.php?t=1624698
    * https://www.webhostingtalk.com/showthread.php?t=1624698&p=9830111#post9830111

    My old benchmarks at https://community.centminmod.com/posts/44838/ revealed Vultr is using Broadwell-EP based cpus at that time

    > During PHP-FPM compile, Vultr sees march=Broadwell so could Vultr for this VPS be using Intel Xeon E5-2600 v4 based processors ? i.e. Xeon E5-2640v4 @2.4Ghz Intel® Xeon® Processor E5-2640 v4 (25M Cache, 2.40 GHz) Product Specifications ? Would explain why even at lower 2.40Ghz vs Linode's 2.80Ghz, OpenSSL/LibreSSL hardware assisted AES-NI and non-AES-NI results were sometimes in Vultr's favour due to improvements in E5-2600v4 Broadwell architecture

    -O3 -m64 -march=broadwell -pipe -gsplit-dwarf -fvisibility=hidden

    Confirmed on my Sydney Vultr VPS

    with CentOS 7.3 64bit with GCC 6.2.1 reports march=native is for Broadwell

    gcc -c -Q -march=native --help=target | grep march
    -march= broadwell

    Can't hide cpu from proper GCC version that supports your CPU :)
  • edited June 2017
    also if you using source compilations as benchmarks be sure to take into account each VPS's disk i/o performance as it's pretty low (relatively) for Vultr VPSes compared to other VPS providers from my tests.
  • edited June 2017
    Thanks for the links.

    Yeah, as I posted, the newer CPUs are Broadwell, but this isn't what is causing the HUGE speed difference, unless there are some different broadwells in the mix... (my sydney broadwell instance is slower than my sydney haswell, but the other broadwells in the test are the same speed as the haswell)

    I forgot to mention. All my first tests stemmed from a snapshot of my sydney instance - this was built 100% from source, with:
    07:32 (1) "~" jamie@catmint% cpu-dmesg
    CPU: Virtual CPU 714389bda930 (2400.11-MHz K8-class CPU)
    Origin="GenuineIntel" Id=0x306c1 Family=0x6 Model=0x3c Stepping=1
    Features=0x783fbff
    Features2=0xfffa3203
    AMD Features=0x28100800
    AMD Features2=0x21
    Structured Extended Features=0x728
    XSAVE Features=0x1
    07:33 (2) "~" jamie@catmint% show-clang-native-cpu
    Clang "native" target for this machine: haswell
    Features enabled (+) : cx16, xsave, bmi2, fsgsbase, avx, popcnt, fma, bmi, aes, rdrnd, sse4.1, sse4.2, avx2, sse, lzcnt, pclmul, f16c, ssse3, mmx, cmov, movbe, xsaveopt, sse2, sse3
    Features disabled (-) : sse4a, avx512bw, tbm, fma4, avx512vl, prfchw, adx, xsavec, avx512cd, avx512pf, rtm, xsaves, avx512er, avx512f, pku, xop, rdseed, hle, sha, avx512dq
  • edited June 2017
    I have to agree with Jamie. It looks like a virtualization setup issue or CPU type and clock frequency are wrongly reported. Nothing else justifies such a huge linear difference in performance.
    Anyway, I've noticed that all new instances chronologically created after this forum thread are as fast as before.
    Maybe Vultr folks keep an eye on this forum... :blush:
  • Well, if someone has made a change in secret, I'd say they were covering up a cockup rather than a con :-)

    ... interesting to know what happens to any of the "slow" instances when powered off and on again!
  • @Athan Just to throw a spanner in the works, a new instance of mine in Miami is 2.394Ghz but multiple perl 5.26 builds too around 6 minutes.. Right in the middle of the 2 previous reported times!

    CPU: Virtual CPU a7769a6388d5 (2394.50-MHz K8-class CPU)
    Origin="GenuineIntel" Id=0x306d2 Family=0x6 Model=0x3d Stepping=2
  • Hm, something's going wrong at Vultr...
  • edited July 2017
    Five new Frankfurt and Amsterdam instances (1/c 2GB and 2/c 4GB) all 2.394GHz and "slow".
    I'm afraid we have to live with it until Vultr cut performance even more in the future... :(
  • Sounds like what you folks are reporting is basic VPS shared hosting contention with some VPS host nodes being more loaded with busy neighbours than other VPS host nodes.

    If you want less contention of VPS host nodes probably need to use Vultr Dedicated cloud or a different web host. Vultr Dedicated Cloud has more dedicated resources and higher clocked 3.4+ Ghz cpus. I did benchmarks at https://community.centminmod.com/threads/vultr-dedicated-cloud-vs-vultr-performance-vps-benchmarks.3163/
  • edited July 2017
    Not at all.

    You'll see from my posts that I originally ran the same test over a 24 hour period on instances in 7 different locations.

    Athan originally posted his tests:

    "Old (cpu id=0x306c1): 5:01
    New (cpu id=0x306d2): 6:53 (38% slower)"

    Whilst I found the CPU id wasn't the trigger, all my results, from all locations reported consistently within a few seconds of either his "old cpu" time, or the "new cpu" time, depending on instance.

    I.E. ALL were consistently around 5 minutes, or all were consistently around 7 minutes.

    The one I did reported a few days later could have been down to anything - I certainly wasn't giving much relevance to that one result.

    But as for the others, I'm not saying this is some intentional vultr conspiracy, but that is nothing to do with host contention.

    Cheers!
  • yeah whatever it is, probably best to test the spun up Vultr before deciding if it's a keeper :)
  • Vultr is mostly great VPS provider, that's why we're here! :)
    Nobody expects dedicated server class performance consistency but imho a decreasing core performance curve over time is rather an alarming sign.
  • edited July 2017
    BTW, Vultr at https://www.vultr.com/benchmarks/ advertise their performance UnixBench-ed at 1400 (1GB instances) and 2200 (2GB instances)

    Unfortunately the FreeBSD port of Unixbench on latest 2GB instances gives only 457, just a tad higher than a cheapo scaleway atom instance :neutral:

    Before Vultr latest virtualization changes, I was able to achieve 950 - 1080 using the same OS / UnixBench version and I'm pretty sure that owners of old 3.xx GHz instances could get even higher scores.

    Dhrystone 2 using register variables 116700.0 24409029.1 2091.6
    Double-Precision Whetstone 55.0 4842.2 880.4
    Execl Throughput 43.0 1572.4 365.7
    File Copy 1024 bufsize 2000 maxblocks 3960.0 256829.0 648.6
    File Copy 256 bufsize 500 maxblocks 1655.0 95345.0 576.1
    File Copy 4096 bufsize 8000 maxblocks 5800.0 350904.0 605.0
    Pipe Throughput 12440.0 88547.2 71.2
    Pipe-based Context Switching 4000.0 35919.3 89.8
    Process Creation 126.0 6036.8 479.1
    Shell Scripts (8 concurrent) 6.0 352.3 587.2
    System Call Overhead 15000.0 1006282.7 670.9
    =========
    FINAL SCORE 457.5
  • @eva2000 Yep! A pain in the arse, but I agree that as things stand, that's the best solution!
  • edited July 2017
    As far as I understand (and I may be wrong), the way Unixbench works is heavly dependet on compiler/OS etc. so can't really be used to accurately compare different OS's, but more appropriately, the same OS on different hardware.

    I totally may be totally thinking of totally something totally else, though, totally!

    Anyway, out of interest, I ran unixbench on 3 of my original instances. This time, it was just a single pass, and the instances - although quite light on resource use - were live. So, these results need to be taken with a pinch of salt, but however:

    FreeBSD: 11.1 (RC1)

    Score 874.8 : 1MB - CPU: Virtual CPU 714389bda930 (3600.16-MHz K8-class CPU) Origin="GenuineIntel" Id=0x306c1 Family=0x6 Model=0x3c Stepping=1
    Score 641.9 : 1MB - CPU: Virtual CPU e7da7129d3ee (2400.08-MHz K8-class CPU) Origin="GenuineIntel" Id=0x306c1 Family=0x6 Model=0x3c Stepping=1
    Score 637.2 : 2MB - CPU: Virtual CPU 714389bda930 (3392.26-MHz K8-class CPU) Origin="GenuineIntel" Id=0x306c1 Family=0x6 Model=0x3c Stepping=1
    As I expected, memory didn't seem to make a difference in these cases.
  • Hate to jump start an older post, but curious if this is still an issue. Has anyone recently ran additional benchmarks to confirm?
Sign In or Register to comment.

Registration Required

A VULTR.com account is required to use the forum. Click here to sign in.

Quick Links