My site will consist of tens of thousands of listings from imported csv files. The files are large (about 90mb each). I tested a 6mb csv file containing 21,000 listings and it took 12 minutes to fully import on Bluehost servers. The process seems to take longer when testing on Godaddy servers.
My question is, do you know what the maximum file size or number of records the plugin can import for the most efficient import process? Have you tested your plugin’s import feature with large csv files and if so, does 12 minutes for a 6mb file with 21,000 listings sound about right or is this amount of import time too high? I’m just trying to ascertain if it’s even worth using the plugin given that the rest of my files are in the 90mb range.
If anyone else out there has experience with importing large csv files of this size, please chime in. Thanks.
There are not limits imposed by us on the directory size because we added new sequential CSV import process that improves performance and reliability especially for large imports. The maximum size for the directory will be driven entirely from your webhost’s resources. If you have a hosting account with sufficient RAM, a reasonable processor, and enough disk space, there’s no practical limit that I’m aware of for BD. I would think the time it is currently taking is reasonable, considering the size of the entries being imported. However, you may try further breaking down the CSV files into smaller versions to see if that helps improve the speed.
Thanks for the response. We have moved to dedicated hosting with InMotion. Our server specs are as follows:
- Intel® Xeon® X3430
- 2.8GHz Turbo 4C/4T
- 8gb RAM
- 500gb SSD
- Software RAID
What’s the largest number of records you tested the plugin with and how long was the import time? My measly 6mb file of 21,000 records just took 35 minutes to process. The plugin created listings for 19,000 of those records and rejected 2,000. I just want to get a good gauge as to how I compare to your results. If 35 minutes is out of the ordinary then there must be something wrong with my files or my WP install.
Import time is not deterministic in an easy-to-follow formula. It depends on: server capacity/traffic, CPU, RAM available, database throughput speed, PHP memory limit per thread, size of the images you’re importing and the hardware your server uses.
Some servers can do 40K records in under 15 minutes, but they may have no images. Some servers struggle with 800 listings with 500K images attached to each. It depends on too many factors. If you have 40K records, break it up into smaller files (especially if you have images in a separate ZIP, which almost everyone forgets about)
The thing about imports is that this isn’t just us shoving data into a table. We’re calling the WP API (because we’re a custom post type, and that’s what we’re supposed to do) for each record, so WP has processing to do on each one. We’re personally tested with 40K records on reasonable hardware and it imports in a reasonable time (I don’t recall the specific number) so we know this works but it may take some time to get it imported.
The 80 min and 40% sounds low but for 40K, again, it depends on a lot of things we can’t easily predict.
I get what you’re saying. But there aren’t any unknown factors associated with my server. I already listed the hardware aspects of my server and it is state of the art with respectable capacity. IN addition, PHP memory limit is set to 999M, No images involved in our directory, there is no traffic, only one other website on server with no visitors. So there’s no competition for resources or database connections, the hardware as mentioned in my previous comment is state of the art and memory is not an issue.
All I was looking for was a baseline standard to compare my speed versus what you got. How does your hardware compare to what I have? You say 80min and 40% sounds low but what exactly does that mean? Is that a low import rate or is that low time for 40%? If I attached one of my CSV files could you test it by importing it and taking note of how long it took?
Please contact us with the WP admin and FTP credentials to your site so we can look at the specifics of this. We need the following information:
– WP admin username
– WP admin password
– WP login URL
– FTP username
– FTP password
– FTP servername/IP address
– Links to this forum thread
Contact us here with that info so we can help better:
Please disregards above reply. It appears that we have not received the information you’ve sent. Kindly resubmit the information using the contact form below:
Sorry for the confusion.
Most Users Ever Online: 197
Currently Browsing this Page:
Guest Posters: 5121
Newest Members:[email protected]
Moderators: BD Support: 0
Administrators: Business Directory Support: 1716