Daemon Generated KML

Talk about random things WiFiDB related, either new things, concepts, or issues to work out in code.
User avatar
pferland
Contributor
Contributor
Posts: 406
Joined: Mon Oct 22, 2007 8:38 am
Location: The Universe
Contact:

Daemon Generated KML

Post by pferland » Fri Aug 28, 2009 1:38 pm

The Daemon Generated KML

This is a feature that I have wanted to implement for some time now.

The Main KML file:
[/wifidb/out/daemon/update.kml]
This is the KML file that you will initially download and open in Google Earth. It has network links to the 'Newest AP" KML file and the daily Export KMZ file.


The Daily Exports KMZ:
Newest KMZ at:
[/wifidb/out/daemon/fulldb.kmz]

Archived at:
[/wifidb/out/daemon/<date(yyyy-mm-dd)>/fulldb.kmz]

The KMZ file is laid out as such:
doc1.kml...................<- The index file for the KMZ, points to the other KML files in the archive
files/full_db.kml..........<- The full DB export (no Labels)
files/full_db_labels.kml..<- The full DB export (w/Labels)
files/daily.kml.............<- The APs that have been imported so far that day (no labels)
files/daily_labels.kml.....<- The APs that have been imported so far that day (w/labels)


Newest AP KML:
[/wifidb/out/daemon/newestAP.kml ]
When a user's scan list from Vistumbler is being imported, all the new APs that have not been imported into the DB yet, are written to a KML file, if they have GPS, so that when you have the update.kml file loaded into Google earth, you can watch the APs appear on the screen as they are imported into WiFiDB.

Of course as always, the cutting edge things im working on are at http://rihq.randomintervals.com/wifidb/
The Update.kml file can be found here: http://rihq.randomintervals.com/wifidb/ ... daemon_kml
by clicking on the "Sample KML file" link


Stay tuned, more updates to this feature are sure to come, the one I'm working on now, is forking the export functions, so that it can use more then just one CPU. of course, if you only have one CPU, then this may actually decrease performance, seeing how there would now be two threads on one CPU instead of one that is serial.
The best acceleration you can get on a Mac is 9.8ms^2

arizonajon
Contributor
Contributor
Posts: 216
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

Daemon Generated KML

Post by arizonajon » Wed Feb 18, 2015 4:27 pm

pferland wrote: The Update.kml file can be found here: http://rihq.randomintervals.com/wifidb/ ... daemon_kml
by clicking on the "Sample KML file" link.
The above link does not seem to work. It times out after a while.

However, this page

https://live.wifidb.net/wifidb/opt/sche ... daemon_kml

does work, and the link "Current WiFiDB Network Link: Download!" at the top of that page to the "update.kmz" file appears to work fine. I am now watching the greater PHX area awaiting the upload and processing of several vs1 files that I collected a week or so ago.

Pretty cool.

Cheers - Jon

arizonajon
Contributor
Contributor
Posts: 216
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

Daemon Generated KML

Post by arizonajon » Wed Feb 18, 2015 4:35 pm

Yet another noob question.

On the page
https://live.wifidb.net/wifidb/opt/scheduling.php

there is the heading
Scheduled Imports
Next Import scheduled on: 2015-02-10 16:56:48 (UTC) 2015-02-10 09:56:48 (UTC-7)

Why are these imports so far in the past? Today is 18 Feb, and the files are being imported and my GE screen is updating as I type. What is the 10 Feb date referring to?

Cheers - Jon

arizonajon
Contributor
Contributor
Posts: 216
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

Daemon Generated KML

Post by arizonajon » Wed Feb 18, 2015 10:15 pm

I think i may have broken the importer function. There's a 2.1 MB file that's been preparing for import since 3pm MST today...

User avatar
pferland
Contributor
Contributor
Posts: 406
Joined: Mon Oct 22, 2007 8:38 am
Location: The Universe
Contact:

Daemon Generated KML

Post by pferland » Wed Feb 18, 2015 11:53 pm

Hello arizonajon,
Fear not, it was not you that broke the importer, just Andrew. He has been working on a new version of the importer that separates out the import and export functions from the daemon that I had initially created and has issues with long running connections to the MySQL database. We will trry and get the import in as soon as possible.

happy war-driving

-Phil
The best acceleration you can get on a Mac is 9.8ms^2

User avatar
ACalcutt
Vistumbler / TechIdiots Admin
Vistumbler / TechIdiots Admin
Posts: 1162
Joined: Sun Oct 21, 2007 6:50 pm
Location: Rutland, MA
Contact:

Daemon Generated KML

Post by ACalcutt » Thu Feb 19, 2015 7:20 am

arizonajon wrote:Yet another noob question.

On the page
https://live.wifidb.net/wifidb/opt/scheduling.php

there is the heading
Scheduled Imports
Next Import scheduled on: 2015-02-10 16:56:48 (UTC) 2015-02-10 09:56:48 (UTC-7)

Why are these imports so far in the past? Today is 18 Feb, and the files are being imported and my GE screen is updating as I type. What is the 10 Feb date referring to?

Cheers - Jon
Hi Jon,

I am in the process of re-working how the schedule page works. It used to be run a daemon that would write the next time, but I modified it to try and fix a few import issues i've been having. right now the imports are running as a cron job every 10 minutes, but i'm not yet writing the next run time to the page. I am still working on this. I plan on updating the way it runs the import more and then updating that scheduling page to give proper stats again

User avatar
ACalcutt
Vistumbler / TechIdiots Admin
Vistumbler / TechIdiots Admin
Posts: 1162
Joined: Sun Oct 21, 2007 6:50 pm
Location: Rutland, MA
Contact:

Daemon Generated KML

Post by ACalcutt » Thu Feb 19, 2015 7:21 am

arizonajon wrote:
pferland wrote: The Update.kml file can be found here: http://rihq.randomintervals.com/wifidb/ ... daemon_kml
by clicking on the "Sample KML file" link.
The above link does not seem to work. It times out after a while.

However, this page

https://live.wifidb.net/wifidb/opt/sche ... daemon_kml

does work, and the link "Current WiFiDB Network Link: Download!" at the top of that page to the "update.kmz" file appears to work fine. I am now watching the greater PHX area awaiting the upload and processing of several vs1 files that I collected a week or so ago.
you found the correct link. the one posted by phil is outdated and no longer exists.

User avatar
ACalcutt
Vistumbler / TechIdiots Admin
Vistumbler / TechIdiots Admin
Posts: 1162
Joined: Sun Oct 21, 2007 6:50 pm
Location: Rutland, MA
Contact:

Daemon Generated KML

Post by ACalcutt » Thu Feb 19, 2015 7:35 am

arizonajon wrote:I think i may have broken the importer function. There's a 2.1 MB file that's been preparing for import since 3pm MST today...
Hmm, looks like that file is getting a parse error. I'll have to look into it when I have more time since i'm heading to work soon

This is the error the file got
[0;31m2015-02-19 12:20:18[1;33m -> [0;31m16
ummm.... thats not supposed to happen... :/
[0;37m
object(ErrorException)#38 (8) {
["message":protected]=>
string(186) "Error parsing File.
'Guest Wi-Fi | Roka Akor|00:11:74:9D:28:50|Wibhu Technologies, Inc.|Open|None|1|802.11n|6|1,2,5.5,11|6,9,12,18,24,36,48,54|62|-64|Infrastructure|Unknown|2033,62,-64'"
["string":"Exception":private]=>
string(0) ""
["code":protected]=>
int(0)
["file":protected]=>
string(48) "/opt/wifidb-0-30/tools/daemon/lib/import.inc.php"
["line":protected]=>
int(231)
["trace":"Exception":private]=>
array(1) {
[0]=>
array(6) {
["file"]=>
string(41) "/opt/wifidb-0-30/tools/daemon/importd.php"
["line"]=>
int(262)
["function"]=>
string(10) "import_vs1"
["class"]=>
string(6) "import"
["type"]=>
string(2) "->"
["args"]=>
array(2) {
[0]=>
string(82) "/srv/www/virtual/live.wifidb.net/wifidb/import/up/27796403_2015-02-06_20-29-20.VS1"
[1]=>
string(11) "arizonajon|"
}
}
}
["previous":"Exception":private]=>
NULL
["severity":protected]=>
int(1)
}

arizonajon
Contributor
Contributor
Posts: 216
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

Daemon Generated KML

Post by arizonajon » Thu Feb 19, 2015 11:24 am

Good morning, Phil and Andrew -

Thanks for the quick and informative responses. I've just uploaded another small file as I had to run to the store a while ago and decided to wardrive it.

The question I have now is in regards to the update.kmz network link.

I have been playing with the WiFiDB Newest AP radio buttons. If I select either of the "fly to" buttons, I have been assuming that as the database gets populated with new APs, the map will then fly to the new AP. I'd also imaging that this means that the database is now populated with all the APs preceding that newest AP, and that I should see those on GE as well. However, that's not how it appears to behave. In fact, sometimes the "fly to" will oscillate back and forth between two APs which are not consecutive, and might be several miles apart.

It's a pretty nice feature, it'd be equally nice to know what the expected behavior is :D

Now on to work as well.

Cheers - Jon

User avatar
pferland
Contributor
Contributor
Posts: 406
Joined: Mon Oct 22, 2007 8:38 am
Location: The Universe
Contact:

Daemon Generated KML

Post by pferland » Thu Feb 19, 2015 11:39 am

[0;31m2015-02-19 12:20:18[1;33m -> [0;31m16
ummm.... thats not supposed to happen... :/
[0;37m
object(ErrorException)#38 (8) {
["message":protected]=>
string(186) "Error parsing File.
'Guest Wi-Fi | Roka Akor|00:11:74:9D:28:50|Wibhu Technologies, Inc.|Open|None|1|802.11n|6|1,2,5.5,11|6,9,12,18,24,36,48,54|62|-64|Infrastructure|Unknown|2033,62,-64'"
["string":"Exception":private]=>
string(0) ""
["code":protected]=>
int(0)
["file":protected]=>
string(48) "/opt/wifidb-0-30/tools/daemon/lib/import.inc.php"
["line":protected]=>
int(231)
["trace":"Exception":private]=>
array(1) {
[0]=>
array(6) {
["file"]=>
string(41) "/opt/wifidb-0-30/tools/daemon/importd.php"
["line"]=>
int(262)
["function"]=>
string(10) "import_vs1"
["class"]=>
string(6) "import"
["type"]=>
string(2) "->"
["args"]=>
array(2) {
[0]=>
string(82) "/srv/www/virtual/live.wifidb.net/wifidb/import/up/27796403_2015-02-06_20-29-20.VS1"
[1]=>
string(11) "arizonajon|"
}
}
}
["previous":"Exception":private]=>
NULL
["severity":protected]=>
int(1)
}

After looking at the error, the source VS1 file, and the import.inc.php file, the string from the VS1 file that is reported in the error :
Guest Wi-Fi | Roka Akor|00:11:74:9D:28:50|Wibhu Technologies, Inc.|Open|None|1|802.11n|6|1,2,5.5,11|6,9,12,18,24,36,48,54|62|-64|Infrastructure|Unknown|2033,62,-64
Is the source of the problem. The VS1 file format uses pipes ( | ) to separate out data columns. The SSID in the row has a pipe in its name. This then makes my import function think there is one extra column when there truly isn't. It probably didn't help that my error message was less than useful when there was an unexpected number of data columns. You will need to manually remove the pipe out of the SSID in the VS1 file and re-import. Andrew and I will try and work something out with Vistumbler that will hopefully negate a manual edit of pips in SSID's.


As for your question about the Newest AP in the Google Earth Export. If the import is running properly then it should go in order to what the list in the VS1 file is. Not sure why it would be jumping back and forth between two APs that are miles apart. But it does look like the import did go well Link

After the AP has been imported and the Newest AP moves onto the next, you will not see the AP until a Daily AP export has been completed. That is expected functionality.

Let me know how it goes.

-Phil
The best acceleration you can get on a Mac is 9.8ms^2

User avatar
ACalcutt
Vistumbler / TechIdiots Admin
Vistumbler / TechIdiots Admin
Posts: 1162
Joined: Sun Oct 21, 2007 6:50 pm
Location: Rutland, MA
Contact:

Daemon Generated KML

Post by ACalcutt » Thu Feb 19, 2015 1:23 pm

I have a guess on why update.kml would be jumping between locations. live.wifidb.net is hosted by 3 servers, so when you go to something on live.wifidb.net it automatically chooses which of the 3 servers to connect to.

There is a sync process that is supposed to keep those kmls the same on each server,but this must not be working. I may not have added this sync to the new export process I created, so i'll have to look at that also.

arizonajon
Contributor
Contributor
Posts: 216
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

Daemon Generated KML

Post by arizonajon » Thu Feb 19, 2015 2:28 pm

Hi Andrew -

That's the great thing about building a system like you and Phil have done - there's always something to tweak... :D :(

I was looking at today's (19 Feb 2015) daily kmz files on GE. I see the APs which I imported today, as well as a few others in the PHX or the southwest AZ area which I know I didn't do. As well, I see 5 APs in Des Moines and 1 in Charlton MA (and I'm especially sure I didn't drive by either of those locations). According to the file import log at WiFiDB, only my files have been imported in the past 24 hours. Interestingly, it appears that all of these anomalous APs have the same MAC address of 00:00:00:00:00:00. For several in AZ, the first active dates are consistent with a wardrive done by someone else back in 2010, but the last updated shows today's date.

Cheers - Jon

User avatar
pferland
Contributor
Contributor
Posts: 406
Joined: Mon Oct 22, 2007 8:38 am
Location: The Universe
Contact:

Daemon Generated KML

Post by pferland » Thu Feb 19, 2015 5:06 pm

The AP locations being way off does make sense. The way that I identify an AP is by hashing the SSID, Mac Address, Channel, Radio type, and a few other unique fields. So if you have an AP that has the same info and since the mac address is all 0's then there is a high chance of an AP already existing with that data. As for the other APs in the Arazona area, a few years ago there was someone else importing from that area.

Andrew has kind of taken over as spiritual leader of the project for now since i have some what a little moved onto other projects. but i do give advice and help him out from time to time and if i do get bored, i do some coding myself, which hasnt been in a while on the wifidb project.
The best acceleration you can get on a Mac is 9.8ms^2

arizonajon
Contributor
Contributor
Posts: 216
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

Daemon Generated KML

Post by arizonajon » Thu Feb 19, 2015 10:59 pm

I appreciate the work that both of you done and the time and effort that must go / have gone into it.

I imported a few more drives from today. For the most part, the importing goes very smoothly. GE crashes ("Google Earth must close..." on occasion when attempting to fly to a location, it seems to have a problem providing me an error log as well, so it might be something with GE.

So far I can't say that I've seen any particular difference in performance after changing the Win 7 performance setting. Loop times are still variable, but often quite long (20 sec). I do believe that the GPS is handling much better, even in just looking at the GE plots and seeing whether APs are bunched into single well separated positions or they're distributed in more positions, suggesting greater position update rate. Obviously, there's the ones which were collected while at a intersection red light, sometimes 20-30 at a single location. This particular GPS does have the annoying position filter built in, so even if I move a few feet at a red light it sometimes won't report an updated position.

It also appears that due to the limitations of my mobile computer, I have to limit the size of my individual wardrive files to about 2k APs or less. The loop time just continues to increase and the lag gets worse and worse. I don't mind, I would rather it to be faster and more granular rather than keeping thousands of APs in the list. I can always review it later.

When the files get imported and the same AP (same mac address) is found, does the importer look at the RSSI historical record and the new record and use that to make the determination on whether or not to update that AP record?

Cheers - Jon

User avatar
ACalcutt
Vistumbler / TechIdiots Admin
Vistumbler / TechIdiots Admin
Posts: 1162
Joined: Sun Oct 21, 2007 6:50 pm
Location: Rutland, MA
Contact:

Daemon Generated KML

Post by ACalcutt » Fri Feb 20, 2015 8:32 am

For google earth I would change the following

- Set you fly-to speed as fast as it can go in (Tools --> Options --> Navigation)
- Disable any unneeded layers. I Usually just keep "Borders and Labels" and maybe "Roads"
- Turn off the tour guide (View -> Tour guide)
- I experimented with using lower "Texture Colors: High Color (16bit)", but i'm not sure it helped any (i was trying to get all the performance i could out of the atom)

Vistumbler and wifidb both will try to place the access point where it had the best wireless signal, then best gps position. in wifidb there should be a historical records of each access point (you can see who imported it previously).

The other person that likely imported in AZ was this user https://live.wifidb.net/wifidb/opt/user ... StephenACS

arizonajon
Contributor
Contributor
Posts: 216
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

Daemon Generated KML

Post by arizonajon » Sat Feb 21, 2015 9:55 am

Hi Andrew -

Thanks for the recommendations.

The home machine (i7-4790, 16 GB RAM) is where I've been using GE, and getting the crashes. It could be a number of things, including video driver issues. I've suffered a variety of those from the first day I ever loaded GE onto a PC...

The mobile machine (D2550 dual-core Atom, 4 GB RAM) is doing only the VS, and once a minute it grabs a picture from the dashcam using YAWCAM. I am avoiding the GE stuff mobile to keep the CPU load to a minimum.

I have VS set to make a sound (the "reverb spring") every loop time where a new AP is recorded (but not for every AP, and not adjusting the volume, etc.). I assume right now that the GPS is updated once every loop time. Is that true? It would be a great help for me to have a sound when the GPS gets updated, because right now I find myself glancing at the screen to see if the position has changed. That can help tell me to go slower. Is that possible to set up?

Cheers - Jon

User avatar
ACalcutt
Vistumbler / TechIdiots Admin
Vistumbler / TechIdiots Admin
Posts: 1162
Joined: Sun Oct 21, 2007 6:50 pm
Location: Rutland, MA
Contact:

Daemon Generated KML

Post by ACalcutt » Sun Feb 22, 2015 8:17 am

Yeah, right now vistumbler tries to update gps every loop. It will only update if it gets a GPGGA sentence from the gps.

It should be simple to add a sound for new gps position. any suggestions on the sound to use?


I don't know if you have seen, but we have also experimented some with taking pictures durring out scans, which is why vistumbler has some experimental camera features. These are some pictures and videos from our attempts

http://www.techidiots.net/pictures/proj ... er-car-cam
https://www.youtube.com/watch?v=QWhR4twojVQ
http://www.techidiots.net/pictures/projects/car-cam-v2
https://www.youtube.com/watch?v=DipRg-rjLvw

arizonajon
Contributor
Contributor
Posts: 216
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

Daemon Generated KML

Post by arizonajon » Sun Feb 22, 2015 12:05 pm

ACalcutt wrote:Yeah, right now vistumbler tries to update gps every loop. It will only update if it gets a GPGGA sentence from the gps.

It should be simple to add a sound for new gps position. any suggestions on the sound to use?

I don't know if you have seen, but we have also experimented some with taking pictures durring out scans, which is why vistumbler has some experimental camera features. These are some pictures and videos from our attempts

http://www.techidiots.net/pictures/proj ... er-car-cam
https://www.youtube.com/watch?v=QWhR4twojVQ
http://www.techidiots.net/pictures/projects/car-cam-v2
https://www.youtube.com/watch?v=DipRg-rjLvw
The camera stuff you're working on is pretty cool. What kind of cameras did you use to make the pano picture? Looks like a 180 deg view? Very low distortion!

I'm using a Mobius ActionCam with 120deg lens mounted immediately in front of the rear-view mirror. You can see the images it captures at my site at www.jonadams.com. I don't have an archive of the images yet.

For GPS position, any reason that only the GGA message is used? In my setup, and for some unknown reason, VS seems to have a real difficulty decoding the GGA but not the RMC sentence. I see RMC messages in the text window all the time but rarely a GGA. Using a serial data trap on the GPS line or using one of the virtual COM port tools, the GGA sentence is there every second, so I really can't explain it.

That having been said, since the GPS read should be very often, a very short unobtrusive "pip" would be best. I don't have a particular sound to offer, but I see (hear) that the "Windows Feed Discovered.wav" sound (FAX menu) is short and sweet.

Cheers - Jon

User avatar
ACalcutt
Vistumbler / TechIdiots Admin
Vistumbler / TechIdiots Admin
Posts: 1162
Joined: Sun Oct 21, 2007 6:50 pm
Location: Rutland, MA
Contact:

Daemon Generated KML

Post by ACalcutt » Sun Feb 22, 2015 2:02 pm

The last video was the best we did. That was this setup http://www.techidiots.net/pictures/projects/car-cam-v2. In that video I used 5 cannon A720 cameras set up in ..half a loop. I was using CHDK firmware (http://chdk.wikia.com/wiki/CHDK) that allowed me to trigger the camera by applying voltage to the usb connection, which I hooked to a usb relay board controlled by vistumbler.

After the run I took the 5 separate camera shots and used Higgin(http://hugin.sourceforge.net/) to splice them all together to make panoramas


Each of the runs we did was a lot of work. Setting up a big rig, doing all the driving, coming back and trying to get all those images spliced back together. I had a lot of issues with timing and getting the photos to actually line up. I had to get each camera to have identical settings to take pictures at the same time when triggered. My camera were getting filled with bugs since I wasn't protecting the lenses in any way.

When it worked right the pictures came out beautiful. They were 5 nice 3264x2448 pixel images stitched together, so the panos came out really nice.

User avatar
ACalcutt
Vistumbler / TechIdiots Admin
Vistumbler / TechIdiots Admin
Posts: 1162
Joined: Sun Oct 21, 2007 6:50 pm
Location: Rutland, MA
Contact:

Daemon Generated KML

Post by ACalcutt » Sun Feb 22, 2015 9:11 pm

Jon,

I put out an update that has an option to give a sound every time there is a GPGGA update. There is a new menu item in the options menu called "Play sound on new GPS"


I was also thinking about your comment in the Mysticache forum that gps worked better in that program. It looks like I am using the CommMG interface by default in that program instead of the Kernel32 I use in vistumbler. I gave both a try with this new gps sound and it did seem like CommMG was updating more frequently. Have you tried out CommMG in (Settings --> GPS Settings) to see if it works any better?

The downside on CommMG is if the gps gets pulled out while vistumbler is using it vistumbler will crash since I had no way to safely disconnect when using that method. CommMG was the old default gps interface before I added Kernel32 (which is built into widows and handled disconnects better)

EDIT: I made a few changes that I think should help the Kernel32 interface method. The main one was to increase the buffer size i was waiting for since i noticed sentances getting cut off. This change is available in v10.6 Beta 6

EDIT 2: If you are running 32bit windows I also recommend trying the netcomm method. Grab the netcomm installer from https://home.comcast.net/~hardandsoftwa ... ommOCX.htm . Look for "Updated installer for Windows 7" (i used this with windows 8.1 x32 and it worked fine). This would work better than CommMG if you are in 32 bit windows since it doesn't cause any stability issues.

arizonajon
Contributor
Contributor
Posts: 216
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

Daemon Generated KML

Post by arizonajon » Mon Feb 23, 2015 5:36 pm

ACalcutt wrote:Jon,

I put out an update that has an option to give a sound every time there is a GPGGA update. There is a new menu item in the options menu called "Play sound on new GPS"
...


Thank you very much for this - I just installed the update. It sounds a little like a Geiger counter :) I will take it out for a spin a bit later and see how it performs! It appears to tick on a successful GPS read, not limited to a new lat/lon. Is that correct?

Continuing on the serial GPS thing, I noticed something very interesting this morning on a trip to downtown and back. I was almost home, so I had accumulated nearly 3k APs, and driving at about 35 mph up a street which has a lot of condos/apts, so it'd be likely to have a bunch of APs as well. In my intermittent glances at the GPS indicator on VS, the GPS text line was showing an RMC sentence with the time somewhere around 45 seconds after the minute. There must have been a very long loop in this area, since the GPS didn't update for 10-15 seconds. However, when it did it showed the RMC for 47 sec after, then 49, then 55, then in rapid succession, while i was stopped at the red light, many more until finally catching up with real time at about 06 seconds after. This progression from 47 before to 06 after was much shorter than 19 seconds.

ACalcutt wrote:I was also thinking about your comment in the Mysticache forum that gps worked better in that program. It looks like I am using the CommMG interface by default in that program instead of the Kernel32 I use in vistumbler. I gave both a try with this new gps sound and it did seem like CommMG was updating more frequently. Have you tried out CommMG in (Settings --> GPS Settings) to see if it works any better?...

The downside on CommMG is if the gps gets pulled out while vistumbler is using it vistumbler will crash since I had no way to safely disconnect when using that method. CommMG was the old default gps interface before I added Kernel32 (which is built into widows and handled disconnects better)

EDIT: I made a few changes that I think should help the Kernel32 interface method. The main one was to increase the buffer size i was waiting for since i noticed sentances getting cut off. This change is available in v10.6 Beta 6.

EDIT 2: If you are running 32bit windows I also recommend trying the netcomm method. Grab the netcomm installer from https://home.comcast.net/~hardandsoftwa ... ommOCX.htm . Look for "Updated installer for Windows 7" (i used this with windows 8.1 x32 and it worked fine). This would work better than CommMG if you are in 32 bit windows since it doesn't cause any stability issues


I believe I'd tried switching the comms method a while back when first starting with VS. I also installed the netcomm method then. As well, I played with both the native COM port on the mainboard and two different chipsetted USB to serial dongles. At that time, I am pretty sure I didn't know what I was looking at, so it would be a good time to revisit. Thanks for the reminder.

Cheers - Jon

User avatar
ACalcutt
Vistumbler / TechIdiots Admin
Vistumbler / TechIdiots Admin
Posts: 1162
Joined: Sun Oct 21, 2007 6:50 pm
Location: Rutland, MA
Contact:

Daemon Generated KML

Post by ACalcutt » Mon Feb 23, 2015 8:34 pm

I made some more fixes to gps today and changed the way the sound is generated. it should now make a sound whenever the gps position changes instead of on GPGGA update. I also had to change the way the sound was played because the way I was using had problems when sounds overlapped...which made it sound kind of weird.

I also made a change to how I am writing updates to the listview (the list of APs) and the treeview. I am now doing batch updates the the listview/treeview, which seems to have helped performance on my atom tablet.

These changes are v10.6 Beta 8

arizonajon
Contributor
Contributor
Posts: 216
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

Daemon Generated KML

Post by arizonajon » Mon Feb 23, 2015 8:53 pm

Good evening, Andrew -

I didn't yet try the new new version which you mention here, but the first new version with the GPS tick sound is good. It's allowing me to gauge without looking how often I get a GPS position. That's very good, and I can see that it varies radically based upon the number of APs encountered per loop. I took a drive to downtown Scottsdale (and while I was at it, visited a nice microbrewery). I ran in commMG mode the whole way. I will try to go between the various serial port modes tomorrow and see if I can perceive a difference.

Cheers - Jon

User avatar
ACalcutt
Vistumbler / TechIdiots Admin
Vistumbler / TechIdiots Admin
Posts: 1162
Joined: Sun Oct 21, 2007 6:50 pm
Location: Rutland, MA
Contact:

Daemon Generated KML

Post by ACalcutt » Mon Feb 23, 2015 9:24 pm

I gave that new GPS sound a test on the way to work and I could see it definitely wasn't giving a sound on every gps update. what I heard varied a lot, but the gps actually seemed to be updating more regularly. Hopefully the new version fixes this.

arizonajon
Contributor
Contributor
Posts: 216
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

Daemon Generated KML

Post by arizonajon » Tue Feb 24, 2015 11:27 am

A weird thing happened to VS.

Yesterday late afternoon, I was running the version which was dated 22 Feb. I took it out for the spin, listened to the ticks as I said above, everything was good. When I returned to the house, I closed VS, it asked whether I wanted to save the file, I said yes, as usual, it did its thing, and then said saved and clicking ok, it exited. I went into the house, opened TeamViewer, and logged into the mobile computer desktop, and could not find the file just saved where all the other VS files get saved. Got distracted, and forgot about it. I then shut down the mobile PC.

This morning, I started up the PC, and then VS. The VS window opened, and now the font color was the same as the background color (black or near black) for the tables and text. By tables, I mean the one which pops up in the "do you want to use one of the previous databases", and then the two which pop up in the main window (the collected AP table window, and the tree to the left which allows selection by SSID, MAC, etc. Oh, and the lat/lon and text sentence windows were invisible as well. However, the buttons Scan APs, Use GPS, Graph 1 and Graph 2 were legible, the menu across the top of the app window, and the name of the Vistumbler file in the title bar. Otherwise, everything was black on black.

I closed VS thinking it was just a startup problem this morning, then noticed a new folder on the desktop labeled "0". Inside it was the VS1 file from yesterday. Odd since I didn't ever create that or save to the desktop.

I rebooted the PC, restarted VS, no difference. Still all black on black. GPS button didn't seem to start the GPS, or at least I didn't hear the tick-tick noise. Opened the GPS settings. Found that the serial port was now set for 4 bits, even parity. Again, I didn't do that. So, I reset it to 4800 8N1, and now I could hear the tick-tick.

I went to the Help menu item and selected "check for updates". It said there was one, so I ran that. The usual window popped up but the text was invisible. After some time, it showed the YES/NO buttons, I selected yes, and it restarted VS but the color problem hadn't gone away. And even weirder, the new update was the file from 2013!

So, I uninstalled VS. Thought it was interesting that when I selected VS from the list of installed programs, then clicked uninstall, VS vanished from the list within a second without any prompts. Not sure if that's usual, but usually I get some kind of prompt.

Reinstalled from the website. Now all appears normal. Am in the process of reconfiguring it to do some testing. Will upgrade to the latest version as well and make sure that the same thing doesn't happen again.

Cheers - Jon

arizonajon
Contributor
Contributor
Posts: 216
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

Daemon Generated KML

Post by arizonajon » Tue Feb 24, 2015 1:28 pm

Just returned from doing a set of road tests to test the various COM methods on a closed circuit here in the neighborhood.

The route is two-ended, cul-de-sac to cul-de-sac, almost exactly 2 miles long, and the speed limit is 20 mph. There is one stop sign, and nearly zero traffic.

First, installed VS again, let it self-update (shows as V10.6 Beta 8, 23 Feb in the title bar). I am using the mobo native COM 1 port for the GPS.

I started each "lap" at the SW end of the neighborhood, closed and started a fresh instance of VS for each run. PC environment was constant otherwise.

Run 1, Kernel32: 214 APs, 160 GPS locations, 6:52 duration
http://jonadams.com/other/2015-02-24%20 ... rnel32.VS1
http://jonadams.com/other/2015-02-24%20 ... rnel32.kml

Run 2, CommMG: 219 APs, 148 GPS locations, 6:44 duration
http://jonadams.com/other/2015-02-24%20 ... CommMG.VS1
http://jonadams.com/other/2015-02-24%20 ... CommMG.kml

Run 3, NetcommOCX: 191 APs, 175 GPS locations, 6:46 duration
http://jonadams.com/other/2015-02-24%20 ... ommOCX.VS1
http://jonadams.com/other/2015-02-24%20 ... ommOCX.kml

On this bit of information, it seems that Kernel32 and CommMG are similar in terms of letting VS do its thing to collect APs. However, NetcommOCX is better to collect GPS positions. Don't know why, or if that is really a valid conclusion based upon one set of data points.

At the end of the runs, I decided to switch to one of the USB to serial COM dongles to see if there was a change in performance based upon those. A couple weeks ago, I had done this and noted really poor performance using the dongles. Here's what I did and here's what happened.

Closed VS. Moved from mobo serial to USB serial on the prolific dongle. Opened VS, configured it to Kernel32, and since I didn't know which of COM3 or COM4 the GPS was on, I first went to COM3. Clicked Use GPS. Nothing showed up on the screen. So I assumed it wasn't COM3. Next, selected COM4. Clicked Use GPS. Suddenly got an error which read something like "Error: variable called does not exist", or something similar. VS crashed. I restarted VS, and the black-on-black coloring is back... Arrggghhh!

So, will uninstall and reinstall VS, and try again.

Cheers - Jon

arizonajon
Contributor
Contributor
Posts: 216
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

Daemon Generated KML

Post by arizonajon » Tue Feb 24, 2015 1:29 pm

oh, and it had the same problems again with COM port settings and the directory on the desktop.

arizonajon
Contributor
Contributor
Posts: 216
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

Daemon Generated KML

Post by arizonajon » Tue Feb 24, 2015 3:45 pm

Second set of runs, identical to the first except that I used an ATEN UC232A Serial to USB adapter on COM4. This uses the Prolific chipset.

Run 2, Kernel32: 194 APs, 162 GPS locations, 6:51 duration, average of 1 GPS location every 2.5 sec.
2015-02-24 CommPortTest_Kernel32_Prolific.VS1
(48.31 KiB) Downloaded 109 times
Run 2, CommMG: 231 APs, 131 GPS locations, 6:49 duration, average of 1 GPS location every 3.1 sec.
2015-02-24 CommPortTest_CommMG_Prolific.VS1
(51.13 KiB) Downloaded 89 times
Run 2, NetcommOCX: 202 APs, 153 GPS locations, 7:06 duration, average of 1 GPS location every 2.8 sec.
2015-02-24 CommPortTest_NetcommOCX_Prolific.VS1
(49.49 KiB) Downloaded 77 times
Then for good measure, I went back and did another run using the original setup of the mobo COM1 and Kernel32.
Run 3, Kernel32: 203 APs, 167 GPS locations, 6:58 duration, average of 1 GPS location every 2.5 sec.
2015-02-24 CommPortTest_Kernel_COM1_control.VS1
(52.97 KiB) Downloaded 80 times
Then, the last thing: I decided to see how many "loops" I had to make to get down to a delta of less than 5 new APs per one-way (1 mile) drive.
Run 3, Kernel32: 280 APs, 658 GPS locations, 38:14 duration, average of 1 GPS location every 3.5 sec (perhaps due to the increased length of the collected AP list?)
2015-02-24 Run3_Control_5.5_Trips_Kernel32_COM1.VS1
(131.21 KiB) Downloaded 79 times
1st Half loop: 178 APs
2nd HL: 211
3rd: 223
4th: 235
5th: 250
6th: 260
7th: 266
8th: 271
9th: 273
10th: 277
1/2 of an 11th: 280 (I was saved by the bell, here, it could have gone to a delta of 5 if I'd driven the rest of the path...)

What does this tell me? That there may be upwards of 290-300 APs along the 1-mile route which are within range of my setup. At least today, with the foliage the way it is, the phase of the moon, color of the sky, etc... It is useful nonetheless.

What this also did was demonstrate to me that there was enough variability in RF reception per run that making any assumption about AP-reception performance based on serial port method based upon a single or even a few drives was not particularly valid. However, in terms of valid GPS positions, it does seem that the Kernel32 stands out in front.

I've really gotta do some day-job work :shock:

Cheers - Jon

User avatar
ACalcutt
Vistumbler / TechIdiots Admin
Vistumbler / TechIdiots Admin
Posts: 1162
Joined: Sun Oct 21, 2007 6:50 pm
Location: Rutland, MA
Contact:

Daemon Generated KML

Post by ACalcutt » Tue Feb 24, 2015 3:59 pm

I think Beta 8 had some issues with the new batch adding to the listview. once the lists got big it started taking forever between loops. I did some more tweaking today so hopefully its better in Beta 9 (have enough updates yet? :D). I'm going to give it a test on the way home from work.
What this also did was demonstrate to me that there was enough variability in RF reception per run that making any assumption about AP-reception performance based on serial port method based upon a single or even a few drives was not particularly valid. However, in terms of valid GPS positions, it does seem that the Kernel32 stands out in front..
I'm surprised by this. In my testing it seemed like CommMG was updating the most frequently. However I may have fixed that with my updates yesterday.
This morning, I started up the PC, and then VS. The VS window opened, and now the font color was the same as the background color (black or near black) for the tables and text.
Usually this happens when the settings file doesn't save properly. I'm not quite sure why this happens sometimes...it is very random. My only solution to this is to delete the [Vistumbler Dir]\Settings\vistumbler_settings.ini and let vistumbler re-create it.
I've really gotta do some day-job work :shock:
same for me. I spent the day today working on Vistumbler instead of my real job. probably should do some real work tomorrow :-)

You sure have been busy. A week in and you already have 1/4 of the access points I have in the database. I guess it helps doing a city to get your numbers up. most of our access points are more rural routes.

arizonajon
Contributor
Contributor
Posts: 216
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

Daemon Generated KML

Post by arizonajon » Tue Feb 24, 2015 9:56 pm

I bet you're all beta'd-out, Andrew!

I found beta 10 ready when I decided to go to the grocery store a little while ago. It installed and the very first thing I noticed was the lack of sound. No GPS tick, no AP sproing. I checked some other sounds that Windows is supposed to make, they all worked, and I ran Winamp and got Internet audio. Rebooted the machine and tried again. Still no audio from VS. I've made sure that all the boxes are checked as before. Did you add a feature that I didn't look at?

On the black on black thing, that's good to know that a full reinstall is not mandatory, that I might be able to just delete the ini file. I'll keep that in mind if it happens again.

Cheers - Jon

User avatar
ACalcutt
Vistumbler / TechIdiots Admin
Vistumbler / TechIdiots Admin
Posts: 1162
Joined: Sun Oct 21, 2007 6:50 pm
Location: Rutland, MA
Contact:

Daemon Generated KML

Post by ACalcutt » Tue Feb 24, 2015 10:05 pm

I found beta 10 ready when I decided to go to the grocery store a little while ago. It installed and the very first thing I noticed was the lack of sound. No GPS tick, no AP sproing. I checked some other sounds that Windows is supposed to make, they all worked, and I ran Winamp and got Internet audio. Rebooted the machine and tried again. Still no audio from VS. I've made sure that all the boxes are checked as before. Did you add a feature that I didn't look at?
I forgot to push out a new file (sounder.exe) I am trying to play the sounds with. you should see another update to download that missing file. once you have that sounds will work again.

arizonajon
Contributor
Contributor
Posts: 216
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

Daemon Generated KML

Post by arizonajon » Wed Feb 25, 2015 3:45 pm

Good afternoon, Andrew -

Thanks for the return of the tick-tick-tick!

I took it for a drive just now and it seems to be fine. Is there anything I should be looking at to provide feedback?

Cheers - Jon

User avatar
ACalcutt
Vistumbler / TechIdiots Admin
Vistumbler / TechIdiots Admin
Posts: 1162
Joined: Sun Oct 21, 2007 6:50 pm
Location: Rutland, MA
Contact:

Daemon Generated KML

Post by ACalcutt » Wed Feb 25, 2015 4:10 pm

Most of the changes I made the last few days were under the hood changes to the listview/treeviee/gps. then fixes to sorting and filtering that broke because of these changes.

The main improvement should be speed when new APs are added into vistumbler. You had mentioned seeing times up to 20 seconds. Are you still seeing high times like that? For me on my tablet I am seeing an average of about 1100ms-2300ms...with some spikes of around 5000ms when a lot of APs are being added.

Other than looking for these speed improvements let me know if you find anything not working. I tested most features I though may be affected by my changes and fixed anything I found was broken.

I'm thinking as long as this release seems stable for about a week of testing it may be time to do a new stable release on the vistumbler site. It looks like the last official release was 3/8/2013, so its probably due for an update that's not a beta :-)

arizonajon
Contributor
Contributor
Posts: 216
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

Daemon Generated KML

Post by arizonajon » Wed Feb 25, 2015 11:29 pm

Hi Andrew -

I took beta 10 out for another spin this afternoon, but was distracted by a number of things so I didn't have enough time to look at it in depth. There was some strangeness in the GPS, but I can't tell whether it was the serial connection or something else. I hope to have some time tomorrow to revisit it.

Cheers - Jon

User avatar
ACalcutt
Vistumbler / TechIdiots Admin
Vistumbler / TechIdiots Admin
Posts: 1162
Joined: Sun Oct 21, 2007 6:50 pm
Location: Rutland, MA
Contact:

Daemon Generated KML

Post by ACalcutt » Thu Feb 26, 2015 12:00 am

Beta 12 is the latest. I released it last night with fixes to filtering. I tested that release today and it seemed to work properly.

arizonajon
Contributor
Contributor
Posts: 216
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

Daemon Generated KML

Post by arizonajon » Mon Mar 02, 2015 9:19 pm

pferland wrote:The Daemon Generated KML

This is a feature that I have wanted to implement for some time now.
...
The KMZ file is laid out as such:
doc1.kml...................<- The index file for the KMZ, points to the other KML files in the archive
files/full_db.kml..........<- The full DB export (no Labels)
files/full_db_labels.kml..<- The full DB export (w/Labels)
files/daily.kml.............<- The APs that have been imported so far that day (no labels)
files/daily_labels.kml.....<- The APs that have been imported so far that day (w/labels)

...
Is the difference between labeled and unlabeled kmz files only supposed to be that the AP icons on GE either have an SSID shown or not?

If so, then why in GE does clicking either DB Export (No Label) or DB Export (Label) get different APs? Maybe the files are built at different times so newer imports might be in one but not the other? I'm looking at APs which were imported over 24 hours ago and comparing the difference between the two.

Cheers - Jon

User avatar
pferland
Contributor
Contributor
Posts: 406
Joined: Mon Oct 22, 2007 8:38 am
Location: The Universe
Contact:

Daemon Generated KML

Post by pferland » Mon Mar 02, 2015 9:57 pm

The files are generated more or less at the same time.

https://github.com/pferland/WiFiDB/blob ... p#L493-502

I am not quite sure why there would be missing AP's.
Could you give a screenshot example in Google Earth?

-Phil
The best acceleration you can get on a Mac is 9.8ms^2

arizonajon
Contributor
Contributor
Posts: 216
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

Daemon Generated KML

Post by arizonajon » Mon Mar 02, 2015 10:11 pm

Hi Phil -

Here you go.

Cheers - Jon

arizonajon
Contributor
Contributor
Posts: 216
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

Daemon Generated KML

Post by arizonajon » Mon Mar 02, 2015 10:18 pm

not sure why the pictures didn't attach - let me try it again. Both screenshots were within about 30 seconds of one another. I wait about 60 seconds after clicking the WIFIDB GE radio buttons.
Attachments
GE_WiFiDB_wLabels.jpg
GE_WiFiDB_wLabels.jpg (612.24 KiB) Viewed 9049 times
GE_WiFiDB_NoLabels.jpg
GE_WiFiDB_NoLabels.jpg (632.71 KiB) Viewed 9049 times

User avatar
pferland
Contributor
Contributor
Posts: 406
Joined: Mon Oct 22, 2007 8:38 am
Location: The Universe
Contact:

Daemon Generated KML

Post by pferland » Mon Mar 02, 2015 11:12 pm

It tried out the update.kmz file on my laptop and the unlabeled Full DB Export took a few good minutes for all the AP's to load up. Maybe its just taking awhile? I couldn't get the networklink Full DB Export Labeled to load. Yet the link would download in the browser and load in google earth on its own. I am even having troubles opening the file. lol

Still looking into things, I am betting the new Google Earth export that I am working on should handle all this data better.

-Phil
The best acceleration you can get on a Mac is 9.8ms^2

arizonajon
Contributor
Contributor
Posts: 216
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

Daemon Generated KML

Post by arizonajon » Wed Mar 04, 2015 2:09 pm

I am watching my GE display right now with the Newest AP w/fly to enabled, and it is moving every minute or so to a new AP in the Worcester area. Andrew's upload is being processed right now and it's neat to see the display moving around with each new AP.

However, when I upload a drive, it rarely moves, and often just is stuck on one AP for a long time. Right now, it moves back and forth between several of Andrew's APs being uploaded in Worcester and every once in a while one of mine, from my upload at 1813 today. This must have something to do with the multiple servers and the synchronization between them?

Cheers - Jon

arizonajon
Contributor
Contributor
Posts: 216
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

Daemon Generated KML

Post by arizonajon » Wed Mar 04, 2015 5:36 pm

I don't know what you guys are doing, but now my latest upload is causing GE to march along moving from new AP to the next. I like that!

User avatar
ACalcutt
Vistumbler / TechIdiots Admin
Vistumbler / TechIdiots Admin
Posts: 1162
Joined: Sun Oct 21, 2007 6:50 pm
Location: Rutland, MA
Contact:

Daemon Generated KML

Post by ACalcutt » Wed Mar 04, 2015 7:22 pm

That is how it is actually supposed to work. When you are importing and have the daemon network link in google earth, the Newest AP should be the latest AP and it will automatically refresh (you can change the rate it refreshes in google earth)

However there is a slight issue with this and the current setup. Wifidb is being load balanced between 3 servers and only one server, the one doing the import, is updating the latest ap kml file. I don't sync that file to the other servers on every update since it updates so often so unless you hit the server doing the import you won't see the latest AP like you are seeing today. What server your computer chooses is kind of luck of the draw...so if that's working today it may not seem to work again tomorrow.

I was just mentioning to phil I may make and api that dynamically generates the latest kml from the database instead of doing file syncing. The database is automatically in sync, so this would make it so no matter what server you hit you would get the latest AP and have it always work as you are seeing today.

User avatar
ACalcutt
Vistumbler / TechIdiots Admin
Vistumbler / TechIdiots Admin
Posts: 1162
Joined: Sun Oct 21, 2007 6:50 pm
Location: Rutland, MA
Contact:

Daemon Generated KML

Post by ACalcutt » Wed Mar 04, 2015 10:56 pm

I created an api like I mentioned, so the newest ap is dynamically generated now

You are going to need the latest "Combined KMZ Network Link" which has the new urls in it.

arizonajon
Contributor
Contributor
Posts: 216
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

Daemon Generated KML

Post by arizonajon » Thu Mar 05, 2015 7:34 pm

ACalcutt wrote:I created an api like I mentioned, so the newest ap is dynamically generated now

You are going to need the latest "Combined KMZ Network Link" which has the new urls in it.
Thanks for this, Andrew.
Are the links for the "Newest" APs supposed to be working? They appear to time out after a few seconds or so with no data received.
Cheers - Jon

User avatar
ACalcutt
Vistumbler / TechIdiots Admin
Vistumbler / TechIdiots Admin
Posts: 1162
Joined: Sun Oct 21, 2007 6:50 pm
Location: Rutland, MA
Contact:

Daemon Generated KML

Post by ACalcutt » Thu Mar 05, 2015 8:16 pm

arizonajon wrote: Are the links for the "Newest" APs supposed to be working? They appear to time out after a few seconds or so with no data received.
Cheers - Jon
It may have been me modifying the new api I created to make it more compatible with android. the api wasn't working for about 10 minutes ~30 mins ago. it should be working now as long as you downloaded that new network link (in the snippet you quoted)

User avatar
pferland
Contributor
Contributor
Posts: 406
Joined: Mon Oct 22, 2007 8:38 am
Location: The Universe
Contact:

Daemon Generated KML

Post by pferland » Sat Mar 07, 2015 9:35 pm

So I have a nice little sample of the new Daemon Generated KML Exports.

I kind of stole one of your imports Jon and used one from a drive I did recently.

The file is
demo.zip
(296.02 KiB) Downloaded 78 times
Extract it to a folder and open the doc.kml with Google Earth.

You will probably be centered over Massachusettes and be able to see the Access Points in that area.

If you zoom out enough, the Access Points will drop out of view. This is expected.

Then you can start to zoom in on Phoenix, Az and will see Access Points from the other sample area. If you cannot see them, you can double click on the network link to the Arizona label.
google_earth.png
google_earth.png (23.26 KiB) Viewed 9001 times
And that is pretty much it.
The best acceleration you can get on a Mac is 9.8ms^2

arizonajon
Contributor
Contributor
Posts: 216
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

Daemon Generated KML

Post by arizonajon » Sun Mar 15, 2015 11:21 pm

Hi Phil -

Thanks for the above. Works just as you say. Might be a nice way to limit the total number of APs that GE tries to render at any one time.

For the past week or so, I've been having GE crash every time I click on the "Current WiFiDB Network Link" on this page

https://live.wifidb.net/wifidb/opt/sche ... daemon_kml

It was working fine when I started playing with this in February, but like I said, now it's not.

Computer is 16 GB i4790 quad-core running Win7Pro with all updates. Click on the link, GE opens, stalls for 3-4 minutes, then starts to zoom in to my usual home position in Arizona. However, as it's doing that, GE crashes and the only options are to close the program and send a report to Google or to just close the program.

Also, on my MacBook Pro GE always crashes on the Full DB. The Mac had only 4 GB of RAM, so I doubled it to 8, but no difference. Gets the "quit unexpectedly" error.

Any ideas?

Cheers - Jon

arizonajon
Contributor
Contributor
Posts: 216
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

Daemon Generated KML

Post by arizonajon » Mon Mar 16, 2015 12:20 am

I've got Windows Task Manager open now to see more about what's going on with GE.

Start GE. CPU usage doesn't change much, bumps from 4-6% to 15-20% for several minutes, during which time GE is showing the box "Loading myplaces.kml, including enabled overlays". GE memory slowly ramps up to about 830 MB without activating any of the links in the WiFiDB AutoKMZ Generation. CPU usage finally drops back to 4-6%.
I can activate a number of other real-time network links in GE and the memory usage slowly rises to about 960 MB. Some of these links are causing many hundreds of objects to get displayed, and updated every 30 seconds. GE stable and responding to mouse promptly. If I turn off those network links, GE memory usage does not drop from the 960 MB value. Let me close and restart GE.
GE now restarting. Have to wait several minutes while it loads. My network usage is near zero, so this may be due to congestion at the GE servers? Unknown. I do see a blip on the networking display when GE finally begins to respond, nwk peak is 0.1% (avg of 0.02%).
Alright. GE back to 830 MB as before. Click on WiFiDB newest AP. RAM increases a few MB to 842. Deselect, doesn't change. Select WiFiDB Daily report no label, memory usage goes up to 868 MB. Even zooming out to see the entire state and my run from yesterday, mouse is still pretty responsive and memory use is stable at 872 MB. Deselect Daily no label, memory usage doesn't drop. Select Daily w/label, memory usage climbs to 926 MB for entire state. Doesn't matter zoom level, always about same memory usage. However, GE noticeably slow now, responds slowly to mouse input. CPU usage stays at 4-6%. Deselect all WiFiDB selections.
Zoomed in to about 1/2 mile diameter around Wittman AZ, where as far as I know there's never been any APs recorded. CPU is at 6%, memory at 942 MB (38%). Click on WiFiDB Full DB Export no label. Memory usage jumps within 20 seconds to 3732 MB (56%), CPU went up to 9-11%, but back to 6%. As soon as I attempt to interact with GE (scroll, click radio buttons, etc.) GE throws fatal error and must close.
Maybe it's my machine, and maybe it's just too many objects for GE to manage. Don't know...

Cheers - Jon

User avatar
ACalcutt
Vistumbler / TechIdiots Admin
Vistumbler / TechIdiots Admin
Posts: 1162
Joined: Sun Oct 21, 2007 6:50 pm
Location: Rutland, MA
Contact:

Daemon Generated KML

Post by ACalcutt » Mon Mar 16, 2015 10:49 am

Really what has happened is we have to many access points for the simple way we are making the kml. Your area is now like ours. you have so many access points google earth struggles to render them.

You mention google earth crashes when you load the latest kml. This is because google earth is struggling to display a full globe of access points. since we have no region/view settings set you can see google earth trying to display every access point even when zooomed all the way out. What I do to view the full db is as soon as I load the update.kml I click the newest AP w/Fly to and google earth usually brings me right to that AP. once zoomed in google earth handles the full db a little better I unclick the latest AP and start moving around from there. You can zoom out from that point, but once a certain number of APs are in view you will see google earth start to stuggle.

I think the solution to this is what Phil has been working on. Setting up region blocks to limit what google earth is trying to display at once. You can see this type of region stuff with googles own placemarks. Google has tons of placemarks in google earth and it doesn't have any trouble...but if you notice they only show when you zoom in (unlike our placemarks that stay visible and get piled on top of each other as you zoom out)

Post Reply