What's Up with WiFiDB?

Talk about random things WiFiDB related, either new things, concepts, or issues to work out in code.
Post Reply
arizonajon
Contributor
Contributor
Posts: 218
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

What's Up with WiFiDB?

Post by arizonajon » Mon Nov 26, 2018 7:56 pm

Hi Phil and Andrew -

I see that the site appears to be going through a complete db rebuild, or something like that. Any idea when it will be done and ready to allow new uploads and show the newest APs in Google Earth?

Cheers - Jon N7UV

User avatar
ACalcutt
Vistumbler / TechIdiots Admin
Vistumbler / TechIdiots Admin
Posts: 1167
Joined: Sun Oct 21, 2007 6:50 pm
Location: Rutland, MA
Contact:

Re: What's Up with WiFiDB?

Post by ACalcutt » Tue Nov 27, 2018 10:32 am

Hi Jon,

At the moment it's just me here, Phil has moved on and I haven't heard from him in over a year, almost 2 now.

I have been working on a rebuild of the wifidb backend. I have changed the backend mysql structure a bit so I can better troubleshoot import issues and to make things a bit more consistent (no more datetimes in 3 different formats, less repeated data in multiple tables). I have also had to do a lot of work on the frontend and apis to get them to work with my new database structure, which is mostly done, but i'm sure there are some things I will find still need fixing.

I have started the rebuild over a few times the last few days because I found some additional things I didn't notice in dev, like the mysql charset we were using wasn't true UTF8, so it wasn't translating certain character correctly. I changed the backend mysql tables to the 'utf8mb4' charset, which is a proper UTF8 (even once fixing that, the import processes wasn't reading UTF8 correctly, which has also been fixed). I also noticed mid import I missed some of the database changes I made in production and had to start over.

So ya, I am doing a rebuild due to the new database structure. Once it completes you should have all your history back and I will turn on imports again. Hopefully the data will be more consistent in the new version, since a lot of improvements have been made to the import process.


Some other changes I have worked on
- Updated to PHP 7.2, the original project was made for PHP 5.3
- Updated to MariaDB v10.3 (a lot of things broke with 10.3 due to our inserts having a blank id field, so suddenly a lot of things started failing silently, because on the backend they were failing to insert)

My version of the WifiDB has been re-homed at https://github.com/acalcutt/WifiDB

-Andrew

User avatar
ACalcutt
Vistumbler / TechIdiots Admin
Vistumbler / TechIdiots Admin
Posts: 1167
Joined: Sun Oct 21, 2007 6:50 pm
Location: Rutland, MA
Contact:

Re: What's Up with WiFiDB?

Post by ACalcutt » Thu Nov 29, 2018 5:06 pm

I had to restart the rebuild again today. I realized it's been reading Network Type incorrectly for the newer VS1 formats. I've been noticing for a while Infrastructure has been coming in as a number, but today I realized is because it was reading the wrong field on import. unfortunately no way to correct it without a re-import from the source files, so i fixed the issue and restarted yet again... stinks because it had already re-imported 1 million access points.

arizonajon
Contributor
Contributor
Posts: 218
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

Re: What's Up with WiFiDB?

Post by arizonajon » Tue Dec 11, 2018 9:05 pm

Good evening, Andrew -
Thanks for the update on the server effort. I see it's still in some intermediate state. How long do you suspect it will be before I can upload some more files? I'm assuming that since the db shows only 2.7 M APs right now, and there were about 3.4 M before this, that eventually the db will be back to the old count.
I've got a new Win10 box to replace the mobile Win7 box. So far, I'm not really happy with the Win10 environment, it seems kinda bloated and, for my tiny mobile screen, really hard to navigate. However, it's a quad-core box at least 2 generations newer than the current box, so it certainly should be faster at scanning.
I've also been running Wireshark to see the protocol working as i encounter Cox Wi-Fi APs driving along the street. At least with the Win7 box, it takes so long to go through the handshakes that I'm usually out of range by the time it's done. Makes me long for the "good ol' days" when that experience seemed to be so much faster.
Cheers and 73 - Jon N7UV

User avatar
ACalcutt
Vistumbler / TechIdiots Admin
Vistumbler / TechIdiots Admin
Posts: 1167
Joined: Sun Oct 21, 2007 6:50 pm
Location: Rutland, MA
Contact:

Re: What's Up with WiFiDB?

Post by ACalcutt » Fri Dec 14, 2018 9:00 pm

We're up to march 2018, so I would suspect I'm almost done with the rebuild...probably by the end of the week. I expect the count will be slightly lower, since one of the issues I fixed was causing duplicate APs (it was reading files fields incorrectly, with one of the fields used in ap hash)

I notice right now it is importing some of your huge files that have basically no new APs, so it is taking forever basically adding huge amounts of signal history. From watching the imports I have the feeling it gets stuck updating the highest GPS when it hits an access point with huge amounts of history...i see the imports pausing on your "cryptic" APs quite often...i think its because it is querying all signal history to figure out its high GPS point every time it sees it, and it just has a ton of history...

I have been working on a lot of front end improvements. I've revived and improved Theme support and am testing a new css version of my outdated vistumbler theme. I may disable this at some point, but right now you can switch between the new theme and the old one in the "Themes" section. I've also added Map, Json, and KMZ icons in a lot of places they didn't exist before (the globe icon now brings you to the map, the red icon is now kmz). Manufacturer has been added pretty much everywhere. Geonames support on the AP page is now working much more simply. I've also been trying to remove a lot of unneeded cruft we're accumulated over the years.

On the backend I updated the import support. The database can now import Vistumbler VS1,VSZ,MDB, WiggleWifi CSV (android), Wardrive DB/DB3 (andorid), Swardriving CSV (android). Wardrive imports are now a direct import and no longer convert to VS1 first... this allows me to keep all the flags that dosen't exist in vistumbler. I wouldn't mind adding support for other applications if you have something that you prefer...at the moment though, vistumbler provides more information than the others. (none of the android exports provide radio type. the import assumes g or n based on frequency)

I don't know if you had a chance to look at any of the new mapbox maps i've added, but I really find the full DB export quite interesting. The amount you've added really is amazing...I never thought we would have any area in that much detail. ( https://live.wifidb.net/wifidb/opt/map. ... =wifidbmap) . I still want to get a few more levels of satellite imagery though, since the road data (Open Street Maps) for phoenix don't seem nearly as good as they are in Massachusetts and at the level you have to zoom to see all the AP detail the satalite imagery is kind of blury.

I've been highly active updating the wifidb ( https://github.com/acalcutt/WifiDB/commits/master ), but i think I have pretty much everything I wanted done for now. at this time i'm just waiting for the rebuild to complete... I think my limitation is the MYSQL server, becasue i've had 8 CPU cores pegged doing the rebuild.

On a hardware note, i had tried a "Vanquisher 8-Inch Industrial Rugged Tablet PC" over the summer. I really wanted to like it but in the end i couldn't keep a stable gps position with the internal gps. i liked it because it was basically an unlocked standard pc in tablet form...but it was a pain to find all the drivers when i reinstalled it. I ended up returning it due to the gps issues... it was to expensive not to have that work.

Post Reply