Log In
Register
Home
Topic RSS
I’m curious – what’s the largest directory that anyone has built using Business Directory Plugin.
I have a site with 6,715 entries. Â
Members
Offline
Hi Robert,
We have seen people using BD for directories ranging from couple hundred listings to thousands of listings.
There is practically no limit but we don’t normally test BD on extremely large setups because most people just aren’t using the plugin that way. However, if you have a decent server and enough memory to support your searches and traffic, I don’t think you would run into any issues.
Offline
I’ll add one more thing to this:
We’ve personally tested a directory where we imported 40,000 listings and it worked fine. With that said, the largest obstacle to a massive data import for people seems to be 1) their hosting server not being “big” enough and 2) their database being too constrained (hosting restricts queries per hour, number of concurrent queries, etc)
My recommendation to anyone doing a “big” import is this:
1) Don’t use shared hosting. It just won’t work
2) Use a server that is “reasonably” sized
3) Make sure that your database isn’t constrained by contacting your host about the restrictions above. Even when you have a “big” server, you often get a shared database and this causes frustrations because it looks like the directory isn’t working when the database keeps falling over instead.
This article is worth a read as well: http://businessdirectoryplugin…..a-problem/
Documentation: http://businessdirectoryplugin.com/docs
Quick Start Guide: http://businessdirectoryplugin.....tart-guide
When you say you imported 40,000 listings and it worked fine, can you recall how long the import process actually took? Â Were you testing an your own local server?
Business Directory Support said
We’ve personally tested a directory where we imported 40,000 listings and it worked fine.
Members
Offline
Unfortunately, I don’t recall the exact time it took for the import but with a reasonable server, you should be able to import large CSV files in few minutes.
Can you give an estimate of minutes? Are we taking 10 minutes or 120 minutes? My 40,000 record file of 4mb in size is still importing which started 80 minutes ago. It’s currently at 40%.Â
To make sure we’re comparing apples to apples, The columns I have set are:
company,address,city,state,zip,category,phone,website,description,username
My dedicated server’s specifications are:
- Intel® Xeon® X3430
- 2.8GHz Turbo 4C/4T
- 8gb RAM
- 500gb SSD
- Software RAID
I’ve set the upper memory limit to 999M and max file upload size to 250M. I have no other plugins installed and I’m running the basic Twenty Fifteen theme. There are no other sites on my dedicated server.
Is my import rate consistent with yours for a csv file containing 40,000 records?
Members
Offline
Import time is not deterministic in an easy-to-follow formula. It depends on: server capacity/traffic, CPU, RAM available, database throughput speed, PHP memory limit per thread, size of the images you’re importing and the hardware your server uses.
Some servers can do 40K records in under 15 minutes, but they may have no images. Some servers struggle with 800 listings with 500K images attached to each. It depends on too many factors. If you have 40K records, break it up into smaller files (especially if you have images in a separate ZIP, which almost everyone forgets about)
The thing about imports is that this isn’t just us shoving data into a table. We’re calling the WP API (because we’re a custom post type, and that’s what we’re supposed to do) for each record, so WP has processing to do on each one. We’re personally tested with 40K records on reasonable hardware and it imports in a reasonable time (I don’t recall the specific number) so we know this works but it may take some time to get it imported.
The 80 min and 40% sounds low but for 40K, again, it depends on a lot of things we can’t easily predict.
This is amazing. I have not traffic and no competition for resources. We’re running a dedicated & optimized server. My records have no images to process. Yet we can’t import 40,000 records in 15 minutes. It’s more like 85-90 minutes. I’d love to see the specifications on the server that’s importing 40,000 records in 15 minutes especially if the WordPress API processes each custom post type one at a time.
During my last test I had a specialist watch the stats of the server in action as the import was taking place. Apparently only one connection to the database is opened rather than multiple connections. Also, only a tiny fraction of server resources were being used during the import. Is there any way you can update the plugin to allow for faster import?
Members
Offline
I completely understand your points but as mentioned, there are a lot of variables involved that we can’t easily predict without taking a direct look at things. It could be possible that your server most likely has a SHARED database server, and that could easily be a bottleneck.
As you’ve send us your site access information, we’ll investigate this further and share our findings with you on email. Thanks!
Members
Offline
I’m sorry, I’ve checked and we haven’t yet received the information that you’ve sent. Can you please resubmit the information using the contact form below:
https://businessdirectoryplugin.com/contact/
Sorry for the confusion.
Most Users Ever Online: 325
Currently Online: anhsieuno, thewarderer
39 Guest(s)
Currently Browsing this Page:
1 Guest(s)
Top Posters:
Member Stats:
Guest Posters: 5302
Members: 573
Moderators: 1
Admins: 1
Forum Stats:
Groups: 1
Forums: 4
Topics: 10364
Posts: 43734
Newest Members:
thewardererModerators: BD Support: 3
Administrators: Business Directory Support: 1720
