Import slows down dramatically at the end

Post Reply
Smithy
Contributor
Contributor
Posts: 28
Joined: Mon Oct 22, 2007 11:12 am
Contact:

Import slows down dramatically at the end

Post by Smithy »

Hello Andrew, quite a while ago I’ve been here.
Still using Vistumber now and than; still the best!

Vistumber v10.6
Tried to import a 711KB .VS1 file (2141 lines import progress window).
It takes less than 10 seconds to import 98% (2100 lines)
It takes more than 28 minutes to import remaining 2% (41 lines)
No idea what’s wrong with it. Never used it before until today.
Attachments
import 98% after 10 seconds
import 98% after 10 seconds
import .VS1file 98%.png (15.96 KiB) Viewed 12131 times
import 100% after more than 28 minutes
import 100% after more than 28 minutes
import .VS1 file 100%.png (16.16 KiB) Viewed 12131 times
2017-01-29 13-06-59.VS1
VS1 file
(710.48 KiB) Downloaded 304 times
User avatar
ACalcutt
Vistumbler / TechIdiots Admin
Vistumbler / TechIdiots Admin
Posts: 1280
Joined: Sun Oct 21, 2007 6:50 pm
Location: Rutland, MA
Contact:

Import slows down dramatically at the end

Post by ACalcutt »

There's a reason for that. At the start of the VS1 file is all the gps history, one gps/timestamp per line which loads quickly. However, when it gets to the access point data all the signal history for one access point is on one line, so the more signal history you have for the access point the longer it takes to load. This is why you will notice when it starts loading access points the Lines/Min starts dropping quickly.

I've gone though this code many times to try and speed it up but I haven't found any way to improve it. I think the layout of the detailed csv is actually quicker to load, but not by much.

One more note about the signal history being all on one line. If you have an AP with lots of signal history (say you left Vistumbler scanning in one spot for a day), you will have one signal/gps pair for each active AP for every second you were scanning. When you go to load this into vistumbler it will take FOREVER. I highly recommend stopping vistumbler when you are parked or sitting in one spot for a long time.
Smithy
Contributor
Contributor
Posts: 28
Joined: Mon Oct 22, 2007 11:12 am
Contact:

Import slows down dramatically at the end

Post by Smithy »

Andrew, I've examined the VS1 file after reading your answer, and I think I understand now more or less.
What is the sense and value of saving the signal history data? It costs lots of time to import the data, but it's nowhere visible.
I could live perfectly with a import without the signal history data.
My only purpose for saving is to check if new acccesspoints have come around my home later (weeks-months) in time.
Maybe an option in "Settings" (leaving out the signal history) is a solution?
arizonajon
Contributor
Contributor
Posts: 285
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

Import slows down dramatically at the end

Post by arizonajon »

Signal history data is very valuable when you're mobile running Vistumbler. It helps to pin down a probable location (or at least the highest signal strength, an analog to proximity) for a given AP. I use Vistumbler and that feature extensively.

For your application, something like Homedale would be excellent. It will log the first and last time it heard an AP, RSSI, channel, mac address, ssid, speed, vendor, all the usual stuff. And, it's free (as in free beer) as well.
User avatar
pferland
Contributor
Contributor
Posts: 406
Joined: Mon Oct 22, 2007 8:38 am
Location: The Universe
Contact:

Import slows down dramatically at the end

Post by pferland »

You can also use the summery csv export in vistumbler to get one signal point per Access Point.
The best acceleration you can get on a Mac is 9.8ms^2
Smithy
Contributor
Contributor
Posts: 28
Joined: Mon Oct 22, 2007 11:12 am
Contact:

Import slows down dramatically at the end

Post by Smithy »

by pferland » Sat Feb 18, 2017 8:56 pm
You can also use the summery csv export in vistumbler to get one signal point per Access Point.
There is a export function for summery csv, but not an import function
import function for detailed csv doesn't import an summery export
User avatar
ACalcutt
Vistumbler / TechIdiots Admin
Vistumbler / TechIdiots Admin
Posts: 1280
Joined: Sun Oct 21, 2007 6:50 pm
Location: Rutland, MA
Contact:

Import slows down dramatically at the end

Post by ACalcutt »

The import for detailed csv is also supposed to import the summary csv, but that seems to be broken. I have to look to see whats going on with that.
Smithy
Contributor
Contributor
Posts: 28
Joined: Mon Oct 22, 2007 11:12 am
Contact:

Import slows down dramatically at the end

Post by Smithy »

ACalcutt 23 Feb 2017: I have to look to see whats going on with that.
Still waiting for a fix :(
User avatar
ACalcutt
Vistumbler / TechIdiots Admin
Vistumbler / TechIdiots Admin
Posts: 1280
Joined: Sun Oct 21, 2007 6:50 pm
Location: Rutland, MA
Contact:

Import slows down dramatically at the end

Post by ACalcutt »

Sorry for the late replay on this, but summary csv import should be fixed in 10.6.4 Beta 5.

The issue was the BTX and OTX values had commas in them, which causes the csv to have the wrong number of rows. What i've done to fix it is wrap the SSID, Manufacturer, Label, BTX, and OTX fields in quotes like I had done with the detailed csv. Now, if you import as a detailed csv it should also be able to read the summary csv like it was supposed to do.

unfortunately the old exported csv files would need a few tweaks (quotes around the btx and otx values) before they could be imported, but going forward the new files should work.
Smithy
Contributor
Contributor
Posts: 28
Joined: Mon Oct 22, 2007 11:12 am
Contact:

Import slows down dramatically at the end

Post by Smithy »

summary csv import should be fixed in 10.6.4 Beta 5.
Thank you for the fix
Smithy
Contributor
Contributor
Posts: 28
Joined: Mon Oct 22, 2007 11:12 am
Contact:

Import slows down dramatically at the end

Post by Smithy »

Seems fix in 10.6.5 is not fully working:
I can export a scan as Summary Comma Delimited file
I can import that file as a Detailed Comma Delimited file
I can continue scanning on the imported file
I can export the new updated scan as Summary Comma Delimited File
I can NOT import the new file as a Detailed Comma Delimited file. it says Progress Done, But doesn't import any lines

I cannot attach the exported file to this post.
Error: Invalid file extension 2017-04-15 13-12-23.csv
Attachments
2017-04-15 13-12-23.csv.txt
(7.46 KiB) Downloaded 311 times
User avatar
ACalcutt
Vistumbler / TechIdiots Admin
Vistumbler / TechIdiots Admin
Posts: 1280
Joined: Sun Oct 21, 2007 6:50 pm
Location: Rutland, MA
Contact:

Import slows down dramatically at the end

Post by ACalcutt »

That's strange. when you export the file the second time it added a second set of quotes around BTX and OTX. the original quotes should have been removed on import.
User avatar
ACalcutt
Vistumbler / TechIdiots Admin
Vistumbler / TechIdiots Admin
Posts: 1280
Joined: Sun Oct 21, 2007 6:50 pm
Location: Rutland, MA
Contact:

Import slows down dramatically at the end

Post by ACalcutt »

Should be fixed in 10.6.4 Beta 7
Smithy
Contributor
Contributor
Posts: 28
Joined: Mon Oct 22, 2007 11:12 am
Contact:

Import slows down dramatically at the end

Post by Smithy »

Thanks.
Seems to work fine now.
Smithy
Contributor
Contributor
Posts: 28
Joined: Mon Oct 22, 2007 11:12 am
Contact:

Import slows down dramatically at the end

Post by Smithy »

27 out of 59 AVP's qualify vistumbler's update.exe and vistumbler_updater.exe as a virus as reported by virustotal.com
I assume they are false positives, but very annoying and irritant on each and every scan or update
You should do something about it.
Thank you
Post Reply