Import slows down dramatically at the end

Smithy
Contributor
Contributor
Posts: 28
Joined: Mon Oct 22, 2007 11:12 am
Contact:

Import slows down dramatically at the end

Postby Smithy » Sun Jan 29, 2017 1:01 pm

Hello Andrew, quite a while ago I’ve been here.
Still using Vistumber now and than; still the best!

Vistumber v10.6
Tried to import a 711KB .VS1 file (2141 lines import progress window).
It takes less than 10 seconds to import 98% (2100 lines)
It takes more than 28 minutes to import remaining 2% (41 lines)
No idea what’s wrong with it. Never used it before until today.
Attachments
import .VS1file 98%.png
import 98% after 10 seconds
import .VS1file 98%.png (15.96 KiB) Viewed 477 times
import .VS1 file 100%.png
import 100% after more than 28 minutes
import .VS1 file 100%.png (16.16 KiB) Viewed 477 times
2017-01-29 13-06-59.VS1
VS1 file
(710.48 KiB) Downloaded 7 times

User avatar
ACalcutt
Vistumbler / TechIdiots Admin
Vistumbler / TechIdiots Admin
Posts: 1156
Joined: Sun Oct 21, 2007 6:50 pm
Location: Rutland, MA
Contact:

Import slows down dramatically at the end

Postby ACalcutt » Thu Feb 16, 2017 2:30 pm

There's a reason for that. At the start of the VS1 file is all the gps history, one gps/timestamp per line which loads quickly. However, when it gets to the access point data all the signal history for one access point is on one line, so the more signal history you have for the access point the longer it takes to load. This is why you will notice when it starts loading access points the Lines/Min starts dropping quickly.

I've gone though this code many times to try and speed it up but I haven't found any way to improve it. I think the layout of the detailed csv is actually quicker to load, but not by much.

One more note about the signal history being all on one line. If you have an AP with lots of signal history (say you left Vistumbler scanning in one spot for a day), you will have one signal/gps pair for each active AP for every second you were scanning. When you go to load this into vistumbler it will take FOREVER. I highly recommend stopping vistumbler when you are parked or sitting in one spot for a long time.

Smithy
Contributor
Contributor
Posts: 28
Joined: Mon Oct 22, 2007 11:12 am
Contact:

Import slows down dramatically at the end

Postby Smithy » Sat Feb 18, 2017 4:15 am

Andrew, I've examined the VS1 file after reading your answer, and I think I understand now more or less.
What is the sense and value of saving the signal history data? It costs lots of time to import the data, but it's nowhere visible.
I could live perfectly with a import without the signal history data.
My only purpose for saving is to check if new acccesspoints have come around my home later (weeks-months) in time.
Maybe an option in "Settings" (leaving out the signal history) is a solution?

arizonajon
Contributor
Contributor
Posts: 213
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

Import slows down dramatically at the end

Postby arizonajon » Sat Feb 18, 2017 8:17 pm

Signal history data is very valuable when you're mobile running Vistumbler. It helps to pin down a probable location (or at least the highest signal strength, an analog to proximity) for a given AP. I use Vistumbler and that feature extensively.

For your application, something like Homedale would be excellent. It will log the first and last time it heard an AP, RSSI, channel, mac address, ssid, speed, vendor, all the usual stuff. And, it's free (as in free beer) as well.

User avatar
pferland
Contributor
Contributor
Posts: 412
Joined: Mon Oct 22, 2007 8:38 am
Location: The Universe
Contact:

Import slows down dramatically at the end

Postby pferland » Sat Feb 18, 2017 8:56 pm

You can also use the summery csv export in vistumbler to get one signal point per Access Point.
The best acceleration you can get on a Mac is 9.8ms^2

Smithy
Contributor
Contributor
Posts: 28
Joined: Mon Oct 22, 2007 11:12 am
Contact:

Import slows down dramatically at the end

Postby Smithy » Sun Feb 19, 2017 11:02 am

by pferland » Sat Feb 18, 2017 8:56 pm
You can also use the summery csv export in vistumbler to get one signal point per Access Point.


There is a export function for summery csv, but not an import function
import function for detailed csv doesn't import an summery export

User avatar
ACalcutt
Vistumbler / TechIdiots Admin
Vistumbler / TechIdiots Admin
Posts: 1156
Joined: Sun Oct 21, 2007 6:50 pm
Location: Rutland, MA
Contact:

Import slows down dramatically at the end

Postby ACalcutt » Thu Feb 23, 2017 1:53 pm

The import for detailed csv is also supposed to import the summary csv, but that seems to be broken. I have to look to see whats going on with that.

Smithy
Contributor
Contributor
Posts: 28
Joined: Mon Oct 22, 2007 11:12 am
Contact:

Import slows down dramatically at the end

Postby Smithy » Wed Mar 15, 2017 1:02 pm

ACalcutt 23 Feb 2017: I have to look to see whats going on with that.


Still waiting for a fix :(

User avatar
ACalcutt
Vistumbler / TechIdiots Admin
Vistumbler / TechIdiots Admin
Posts: 1156
Joined: Sun Oct 21, 2007 6:50 pm
Location: Rutland, MA
Contact:

Import slows down dramatically at the end

Postby ACalcutt » Sun Apr 09, 2017 8:52 am

Sorry for the late replay on this, but summary csv import should be fixed in 10.6.4 Beta 5.

The issue was the BTX and OTX values had commas in them, which causes the csv to have the wrong number of rows. What i've done to fix it is wrap the SSID, Manufacturer, Label, BTX, and OTX fields in quotes like I had done with the detailed csv. Now, if you import as a detailed csv it should also be able to read the summary csv like it was supposed to do.

unfortunately the old exported csv files would need a few tweaks (quotes around the btx and otx values) before they could be imported, but going forward the new files should work.

Smithy
Contributor
Contributor
Posts: 28
Joined: Mon Oct 22, 2007 11:12 am
Contact:

Import slows down dramatically at the end

Postby Smithy » Thu Apr 13, 2017 8:21 am

summary csv import should be fixed in 10.6.4 Beta 5.


Thank you for the fix

Smithy
Contributor
Contributor
Posts: 28
Joined: Mon Oct 22, 2007 11:12 am
Contact:

Import slows down dramatically at the end

Postby Smithy » Sat Apr 15, 2017 2:31 pm

Seems fix in 10.6.5 is not fully working:
I can export a scan as Summary Comma Delimited file
I can import that file as a Detailed Comma Delimited file
I can continue scanning on the imported file
I can export the new updated scan as Summary Comma Delimited File
I can NOT import the new file as a Detailed Comma Delimited file. it says Progress Done, But doesn't import any lines

I cannot attach the exported file to this post.
Error: Invalid file extension 2017-04-15 13-12-23.csv
Attachments
2017-04-15 13-12-23.csv.txt
(7.46 KiB) Downloaded 7 times

User avatar
ACalcutt
Vistumbler / TechIdiots Admin
Vistumbler / TechIdiots Admin
Posts: 1156
Joined: Sun Oct 21, 2007 6:50 pm
Location: Rutland, MA
Contact:

Import slows down dramatically at the end

Postby ACalcutt » Mon Apr 17, 2017 7:41 am

That's strange. when you export the file the second time it added a second set of quotes around BTX and OTX. the original quotes should have been removed on import.

User avatar
ACalcutt
Vistumbler / TechIdiots Admin
Vistumbler / TechIdiots Admin
Posts: 1156
Joined: Sun Oct 21, 2007 6:50 pm
Location: Rutland, MA
Contact:

Import slows down dramatically at the end

Postby ACalcutt » Mon Apr 17, 2017 8:14 am

Should be fixed in 10.6.4 Beta 7

Smithy
Contributor
Contributor
Posts: 28
Joined: Mon Oct 22, 2007 11:12 am
Contact:

Import slows down dramatically at the end

Postby Smithy » Thu Apr 20, 2017 4:37 am

Thanks.
Seems to work fine now.

Smithy
Contributor
Contributor
Posts: 28
Joined: Mon Oct 22, 2007 11:12 am
Contact:

Import slows down dramatically at the end

Postby Smithy » Fri May 05, 2017 11:16 am

27 out of 59 AVP's qualify vistumbler's update.exe and vistumbler_updater.exe as a virus as reported by virustotal.com
I assume they are false positives, but very annoying and irritant on each and every scan or update
You should do something about it.
Thank you


Return to “Report Bugs”

Who is online

Users browsing this forum: No registered users and 1 guest