WiFiDB C# Client and Up-Loader

Talk about random things WiFiDB related, either new things, concepts, or issues to work out in code.
Post Reply
User avatar
pferland
Contributor
Contributor
Posts: 406
Joined: Mon Oct 22, 2007 8:38 am
Location: The Universe
Contact:

WiFiDB C# Client and Up-Loader

Post by pferland » Sat Jan 23, 2016 9:40 pm

I have been working on a C# up-loader and Client for the WiFiDB.

Right now I have been putting my efforts into the Up-loader for now.
I think this would be a more stable and maybe better performing and less resource hungry (although the AutoIt Up-loader doesn't use that much resources really.)

Currently my wish list of features are:

Up-Loader:
  • Upload Single File
  • Upload folder that has VS1, VSZ, CSV, and DB3 (Wardrive4 Android) files
  • Monitor Imports as they go
  • Move imported files to a folder structure similar to how Andrew is doing with his up-loader.
  • Tab with [Import/Export] daemon status updates.
WiFiDB Client:
  • Wireless scanning like Vistumbler (this is going to take a while, going to wait till all the other features for the WiFiDB are done first.
  • Tab with [Import/Export] daemon status updates.
  • Get Scheduled imports status including attempted bad imports.
  • Browse Imported data.
  • Browse by User, Import, AP, area around GPS Point
  • More as I think of them...
Ambitious I know, it is going to take some time as I am learning C# as I go :dork:

Oh yeah, screenshots of what I have so far:
Untitled0.png
While Uploading Files from a Folder
Untitled0.png (54.98 KiB) Viewed 7432 times
Untitled.png
After Uploaded and Waiting for Daemon to start Import
Untitled.png (91.79 KiB) Viewed 7432 times
Untitled2.png
During Import of File
Untitled2.png (78.84 KiB) Viewed 7432 times
Untitled3.png
Continuing on as the daemon reports it is going along...
Untitled3.png (69.95 KiB) Viewed 7432 times
I think that is about it.
The best acceleration you can get on a Mac is 9.8ms^2

arizonajon
Contributor
Contributor
Posts: 216
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

WiFiDB C# Client and Up-Loader

Post by arizonajon » Wed Jan 27, 2016 12:11 pm

Very impressive, Phil!

I look forward to this becoming a reality.

And congrats on taking on C# - I never even opened a book on that...

Cheers - Jon N7UV

User avatar
pferland
Contributor
Contributor
Posts: 406
Joined: Mon Oct 22, 2007 8:38 am
Location: The Universe
Contact:

WiFiDB C# Client and Up-Loader

Post by pferland » Wed Jan 27, 2016 5:33 pm

Thanks Jon,
I have been wanting to learn C# for some time now, but never was able to think of a good project to use it on. I also have been wanting to make a WiFiDB Thick client for a little while, but was never able to figure out a way to make it or how I wanted it laid out.
I never opened a book myself either. just using TechNet and Google. I do have a C# book, which I probably should read for better (proper) ways of doing things in C#, but ehh things are working along for now. :lol:

I did some more work on the Uploader over the weekend and got the second tab working.
The second tab is to view the current status of the Import/Export daemons on the nodes.
Its still crude, I have only done simple tests with my dev server. I might try uploading the APIv2 code for it to the prod server so I can see how it acts with the prod daemons being spun up and killed per import.

Daemon Tab with Dev Server "Daemons"
daemon_tab.png
The best acceleration you can get on a Mac is 9.8ms^2

User avatar
pferland
Contributor
Contributor
Posts: 406
Joined: Mon Oct 22, 2007 8:38 am
Location: The Universe
Contact:

WiFiDB C# Client and Up-Loader

Post by pferland » Mon Feb 08, 2016 5:46 pm

So I have been working on the settings code for the Uploader for about the last week.
SettingsFormDropDown.png
Settings From Drop Down, showing servers.
SettingsFormDropDown.png (11.22 KiB) Viewed 7357 times
EditServerForm.png
Edit Server Properties From
EditServerForm.png (16.44 KiB) Viewed 7357 times
AddServerForm.png
Add Server to List Form
AddServerForm.png (14.6 KiB) Viewed 7357 times
SettingsForm.png
Server Selection Form
SettingsForm.png (10.09 KiB) Viewed 7357 times
The Selected Server is set as the Active Server that the uploader will connect to.
When you remove a server, you will need to reset the active server.
Settings Menu.png
Settings Menu in the Main UI
Settings Menu.png (57.81 KiB) Viewed 7357 times
AutoSettingsForm.png
Auto Settings Form
AutoSettingsForm.png (15.87 KiB) Viewed 7357 times
ImportSettingsForm.png
Import Settings From
ImportSettingsForm.png (15.45 KiB) Viewed 7357 times
More to come. Getting closer to a public beta soon.

-Phil
The best acceleration you can get on a Mac is 9.8ms^2

User avatar
pferland
Contributor
Contributor
Posts: 406
Joined: Mon Oct 22, 2007 8:38 am
Location: The Universe
Contact:

WiFiDB C# Client and Up-Loader

Post by pferland » Tue Feb 09, 2016 11:34 pm

Got the Auto Import On open working. Also got the Archive Imported Files working.
The Auto Import will only use the Default Import Values for the time being.
I am thinking about making it have the option to have a CSV file with all the notes and titles for each import. If a file is not in the CSV then it will use the default import values.
Going to work on a timer option for the Auto Import, so that you can have the option to close the program after a set time, or close the window and it will stay open.

Still working on the Non-default mode. Not sure how I want to do the Manual Folder import, If I want to have a pop up for each file, or just one for the whole folder. Whole Folder sounds more sane.
Made the Up-loader also check for a selected server before starting the background threads to check for Daemon stats and Import updates.

Still want to have a SQLite database file to keep a history of all the imported files data and reload that each time.

-Phil
The best acceleration you can get on a Mac is 9.8ms^2

User avatar
pferland
Contributor
Contributor
Posts: 406
Joined: Mon Oct 22, 2007 8:38 am
Location: The Universe
Contact:

WiFiDB C# Client and Up-Loader

Post by pferland » Fri Feb 12, 2016 4:26 pm

So I think the Uploader is ready for its initial release. I am just going over a few last things and making sure its semi-tight. Will upload a zip soon.
The best acceleration you can get on a Mac is 9.8ms^2

User avatar
pferland
Contributor
Contributor
Posts: 406
Joined: Mon Oct 22, 2007 8:38 am
Location: The Universe
Contact:

WiFiDB C# Client and Up-Loader

Post by pferland » Sat Feb 13, 2016 1:54 am

After Andrew did some testing of the uploader, there are some bugs that need some extra work. Monday looks like the soonest for a release. Will keep updating. Saturday and Sunday are looking a little full, but will see.
The best acceleration you can get on a Mac is 9.8ms^2

arizonajon
Contributor
Contributor
Posts: 216
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

WiFiDB C# Client and Up-Loader

Post by arizonajon » Sat Feb 13, 2016 11:21 am

I will be exploring the Bayonne Bridge (NJ/NY) this next week for work. I am going to try to put together a better mobile wardriving kit than the current one, which requires me to use the internal Wi-Fi of the notebook, limiting my reception...

User avatar
pferland
Contributor
Contributor
Posts: 406
Joined: Mon Oct 22, 2007 8:38 am
Location: The Universe
Contact:

WiFiDB C# Client and Up-Loader

Post by pferland » Mon Feb 22, 2016 4:10 pm

So work is slowly going along. Came across a performance issue that I would like to at least semi-iron out before release.
At least its not a common issue. I have only come across it when I have ~600+ waiting imports and ~2000 total lines in the Listview for imports.

Pretty sure this has to do with the background threads returning their information for the UI to update, but there is too much returning all at once for the UI to do it quick enough.

Thinking.... :crazy:
The best acceleration you can get on a Mac is 9.8ms^2

User avatar
pferland
Contributor
Contributor
Posts: 406
Joined: Mon Oct 22, 2007 8:38 am
Location: The Universe
Contact:

WiFiDB C# Client and Up-Loader

Post by pferland » Fri Feb 26, 2016 10:27 am

Slowly making progress on the Uploader. Fixed a bunch more bugs last night. There is a few more tests I want to run before making a release.

This is taking much longer than expected, but at least it should be somewhat stable and bug free once its ready.... hopefully... lol

-Phil
The best acceleration you can get on a Mac is 9.8ms^2

arizonajon
Contributor
Contributor
Posts: 216
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

WiFiDB C# Client and Up-Loader

Post by arizonajon » Fri Feb 26, 2016 7:45 pm

looking forward to the hatching

User avatar
pferland
Contributor
Contributor
Posts: 406
Joined: Mon Oct 22, 2007 8:38 am
Location: The Universe
Contact:

WiFiDB C# Client and Up-Loader

Post by pferland » Sat Feb 27, 2016 1:15 pm

Think I have all the major bugs squished. Just doing some performance tuning now. Hopefully no new bugs pop up. *waves fist*
The best acceleration you can get on a Mac is 9.8ms^2

User avatar
pferland
Contributor
Contributor
Posts: 406
Joined: Mon Oct 22, 2007 8:38 am
Location: The Universe
Contact:

WiFiDB C# Client and Up-Loader

Post by pferland » Sat Feb 27, 2016 5:11 pm

The one known issue that I have found is that on upload of an import if the file is too large and dropped by the HTTP server then the API will respond with No Upload File Found. You should not get this on the Prod Server (live.wifidb.net) since I am sure the Upload limit on there is large enough. I just have it limited on my Dev server.

I went though a folder that we have of all the VS1 files ever imported and it has about 8000 files (many many of them are duplicates) the Uploader then was able to upload the ~3500 unique Imports in about 2 Hours 20 Minutes. This was with Trace logging and Debug console write enabled.

It is also without a tweak that I think will help with another known issue. Once there is a considerable number of uploads in the listview the UI becomes slow to respond to commands. This is because the background thread that checks for updates to the uploaded imports has as many things to check and return as there are rows. So basically if there is 1000 rows in the Uploaded Files listview, then the background thread spins off 1000 checks for each row. The checks then send an event to the Main UI to update their row. Now the background thread did not wait or anything, it just returned the data as fast as it was able to get it. so you end up with 1000 events being triggered and then updating the Listview.

I added a random thread sleep in the background thread for each upload update so that there is a random delay with the update coming back tot he Main UI to update the Uploaded Files listview, there still is some delays to the UI but not nearly as bad as it was before. The downside of this is that the rows will update in random order not sequential. I am still going to see if there is a better way to do the updates, but this is the best solution I can think of at this time.

The benchmark of the Uploader with the thread sleep on the background thread and with Trace Logging and debug on the same source folder is about 1 Hour and 10 Minutes with the 3500 imports.

Next is the "Release" Compile with Trace logging and Debug turned off.
That was able to import 3500 files in about 40 Minutes.

So at this point I would say that the WiFiDB Uploader is ready for release. I am going to package it up and post a link soon.

-Phil
The best acceleration you can get on a Mac is 9.8ms^2

User avatar
pferland
Contributor
Contributor
Posts: 406
Joined: Mon Oct 22, 2007 8:38 am
Location: The Universe
Contact:

WiFiDB C# Client and Up-Loader

Post by pferland » Sat Feb 27, 2016 5:45 pm

So here it is Version 1.0 of the WiFiDB Uploader:

https://github.com/pferland/WiFiDBClien ... s/tag/V1.0
The best acceleration you can get on a Mac is 9.8ms^2

arizonajon
Contributor
Contributor
Posts: 216
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

WiFiDB C# Client and Up-Loader

Post by arizonajon » Sat Feb 27, 2016 7:51 pm

Hi Phil -

I downloaded this and unpacked the files into my mobile PC running Win7Pro 32 bit. I put the files in the program files directory under WDBUploader. I double-clicked on the .exe file and after a few seconds, it reported "this program has stopped working" with little information. So, I retried starting it. Another crash. After that, it no longer reported "this program has stopped working" but at the same time it no longer seems to do anything as I then accidentally hit "send this to Microsoft" rather than closing it.

I rebooted the PC just in case, but it didn't make any difference in terms of getting the program to run. However, when I watch the processes, I see two start, one of which is WDBUploader.exe and the other is WerFault.exe. I suspect werfault.exe is the thing that sends the error log to Microsoft. I went out to the truck to kill the network connectivity and try again, but the only thing that changed was I don't see the Werfault.exe process.

Inside the house, I unpacked the files on my home PC, running Win7Pro 64 bit. Set up a directory and placed the files there. I double-clicked on the .exe and I get a WiFiDB Uploader V1.0 window and two tabs, one called Uploaded Files and the other Daemon Statuses. Haven't gone any further as this machine doesn't have a wireless network adapter... But it did indicate to me that the s/w does run, I just don't know why it crashes in my mobile setup.

The only significant difference besides 32 vs 64 bit is that my mobile setup has a small video screen (800x600) and I've actually seen some software not like that (like Win10 of all things). But on the big box inside the house I scaled the window down to a small size and it had no issues.
Any ideas? I'm not sure even how to get back to before the "send to Microsoft" so I can at least see the "program has stopped working" box...

Somewhat uncheerfully, Jon N7UV :D

arizonajon
Contributor
Contributor
Posts: 216
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

WiFiDB C# Client and Up-Loader

Post by arizonajon » Sat Feb 27, 2016 7:58 pm

The plot thickens.

Just installed it on my work notebook, Win7Pro 64 bit, and get the same error as on my mobile PC. This time I was wise enough not to click "want microsoft to help you" button :) :)

User avatar
pferland
Contributor
Contributor
Posts: 406
Joined: Mon Oct 22, 2007 8:38 am
Location: The Universe
Contact:

WiFiDB C# Client and Up-Loader

Post by pferland » Sat Feb 27, 2016 8:00 pm

Mind posting a screen shot of the error?

Interesting that you get an error, last time andrew was able to run it. Wonder if it has something to do with the way I compiled it.
The best acceleration you can get on a Mac is 9.8ms^2

User avatar
pferland
Contributor
Contributor
Posts: 406
Joined: Mon Oct 22, 2007 8:38 am
Location: The Universe
Contact:

WiFiDB C# Client and Up-Loader

Post by pferland » Sat Feb 27, 2016 8:12 pm

Interesting I am also using Win7Pro x64 for my dev machine. Andrew is running Win10 x64 for the few tests that he has run for me.

Do you have .Net 4.5 installed? I know I did compile against .Net 4.5.2
The best acceleration you can get on a Mac is 9.8ms^2

arizonajon
Contributor
Contributor
Posts: 216
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

WiFiDB C# Client and Up-Loader

Post by arizonajon » Sat Feb 27, 2016 8:43 pm

screen shot from work notebook PC
ScreenGrab_WDBUploader.PNG
ScreenGrab_WDBUploader.PNG (247.4 KiB) Viewed 7257 times
Under Control Panel > All Control Panel Items > Programs and Features:
Work notebook is .net 4.5.2
Home PC is .net 4.5.2
Mobile PC .net 4.6.1

User avatar
pferland
Contributor
Contributor
Posts: 406
Joined: Mon Oct 22, 2007 8:38 am
Location: The Universe
Contact:

WiFiDB C# Client and Up-Loader

Post by pferland » Sat Feb 27, 2016 8:51 pm

hmmm interesting, could you upload those mdmp files for me?
The best acceleration you can get on a Mac is 9.8ms^2

User avatar
pferland
Contributor
Contributor
Posts: 406
Joined: Mon Oct 22, 2007 8:38 am
Location: The Universe
Contact:

WiFiDB C# Client and Up-Loader

Post by pferland » Sat Feb 27, 2016 8:54 pm

I was just thinking, yopu are running it in the Program Files folder, Windows requires admin privs to write to that path, so its probavbly denying the SQLite class from writing the DB folder. Maybe try and run it with admin privs?

-Phil
The best acceleration you can get on a Mac is 9.8ms^2

arizonajon
Contributor
Contributor
Posts: 216
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

WiFiDB C# Client and Up-Loader

Post by arizonajon » Sat Feb 27, 2016 9:23 pm

Work notebook:
Just tried running it with admin privileges - no joy.
Moved the whole directory to My Documents folder and tried from there. No joy either.
Same error each time, and same as the first time.

User avatar
pferland
Contributor
Contributor
Posts: 406
Joined: Mon Oct 22, 2007 8:38 am
Location: The Universe
Contact:

WiFiDB C# Client and Up-Loader

Post by pferland » Sat Feb 27, 2016 9:27 pm

hmm interesting. Would you be able to upload the mdmp for me to take a look at?
The best acceleration you can get on a Mac is 9.8ms^2

arizonajon
Contributor
Contributor
Posts: 216
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

WiFiDB C# Client and Up-Loader

Post by arizonajon » Sat Feb 27, 2016 9:40 pm

Could be challenging. The program runs only a few seconds before crashing. Let's see.

Ok, I was able to do a dump but it's 98 MB in size, and 43 MB zipped. Seems kind of large...

Do you have a way for me to transfer a 43 MB file to you?

arizonajon
Contributor
Contributor
Posts: 216
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

WiFiDB C# Client and Up-Loader

Post by arizonajon » Sat Feb 27, 2016 10:05 pm

tried it as well on the mobile PC, but since I accidentally clicked "send to MS", it closes before I can get to it to dump.

User avatar
pferland
Contributor
Contributor
Posts: 406
Joined: Mon Oct 22, 2007 8:38 am
Location: The Universe
Contact:

WiFiDB C# Client and Up-Loader

Post by pferland » Mon Mar 28, 2016 2:42 pm

The dumps you sent me both say that the SQLite dll was not able to be loaded. The only thing I can think of is something is blocking it from being loaded since it is working on your other laptop.

Can you check and see if the SQLite dll properties have the Unblock button like in the image below?
http://i.imgur.com/C9at7ml.jpg


Thanks!

-Phil
The best acceleration you can get on a Mac is 9.8ms^2

arizonajon
Contributor
Contributor
Posts: 216
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

WiFiDB C# Client and Up-Loader

Post by arizonajon » Thu Mar 31, 2016 8:44 am

Hi Phil -

Glad to help - here is a pic of the .dll properties.
WDBUploader_WDBSQLite_properties_workPC.PNG
After I clicked "unblock", I tried running WDBUploader again. The following is what I got.
WDBUploader_WDBSQLite_Allowed.PNG
I can send you the dump file if you want it.

Cheers - Jon N7UV

User avatar
pferland
Contributor
Contributor
Posts: 406
Joined: Mon Oct 22, 2007 8:38 am
Location: The Universe
Contact:

WiFiDB C# Client and Up-Loader

Post by pferland » Thu Mar 31, 2016 9:47 am

Hmm I do t see any images, it is just a filename.
Dumps will help, could you also for giggles check the other dlls that are in the uploaded folder and make sure they are not blocked either?

Thank you!

-Phil
The best acceleration you can get on a Mac is 9.8ms^2

arizonajon
Contributor
Contributor
Posts: 216
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

WiFiDB C# Client and Up-Loader

Post by arizonajon » Fri Apr 01, 2016 7:22 am

sorry about that. Should have checked. Here's the images.
WDBUploader_WDBSQLite_properties_workPC.PNG
WDBUploader_WDBSQLite_properties_workPC.PNG (36.22 KiB) Viewed 6704 times
and the crash.
ScreenGrab_WDBUploader.PNG
ScreenGrab_WDBUploader.PNG (247.4 KiB) Viewed 6704 times
I did check the properties of each file in the directory. They were all in "Block" mode, so I switched them all to unblock. Tried to run it and same crash as usual. I've added the dump to the Drive directory.

arizonajon
Contributor
Contributor
Posts: 216
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

WiFiDB C# Client and Up-Loader

Post by arizonajon » Fri Apr 01, 2016 7:24 am

To be clear, I pushed the "unblock" button and applied it for each file where there was an "Unblock" button.

User avatar
pferland
Contributor
Contributor
Posts: 406
Joined: Mon Oct 22, 2007 8:38 am
Location: The Universe
Contact:

WiFiDB C# Client and Up-Loader

Post by pferland » Fri Apr 01, 2016 8:23 am

Thanks Jon. Will check out the dump sometime today.
The best acceleration you can get on a Mac is 9.8ms^2

User avatar
pferland
Contributor
Contributor
Posts: 406
Joined: Mon Oct 22, 2007 8:38 am
Location: The Universe
Contact:

WiFiDB C# Client and Up-Loader

Post by pferland » Fri Apr 01, 2016 10:56 am

Not sure why I didn't notice this before, but do you have the x86 version of the Microsoft Visual C++ 2010 Redistributable Package installed?

That is required for the SQLite dll to work.
The best acceleration you can get on a Mac is 9.8ms^2

arizonajon
Contributor
Contributor
Posts: 216
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

WiFiDB C# Client and Up-Loader

Post by arizonajon » Mon Apr 04, 2016 9:03 pm

image001.png
image001.png (63.73 KiB) Viewed 6691 times
screengrab. Looks like it's installed?

arizonajon
Contributor
Contributor
Posts: 216
Joined: Wed Feb 04, 2015 11:17 pm
Contact:

WiFiDB C# Client and Up-Loader

Post by arizonajon » Tue Apr 19, 2016 9:15 pm

success!

I collected some files today and used the new uploader to send them. Seemed to work ok.
WiFiDBUploader_19Apr16.png
WiFiDBUploader_19Apr16.png (34.31 KiB) Viewed 6458 times

User avatar
pferland
Contributor
Contributor
Posts: 406
Joined: Mon Oct 22, 2007 8:38 am
Location: The Universe
Contact:

WiFiDB C# Client and Up-Loader

Post by pferland » Tue Apr 19, 2016 10:51 pm

Awesome! Now I need to get to adding those new features in and writing a beginners guide to using the uploader. Documentation is not my strong suit though lol.
The best acceleration you can get on a Mac is 9.8ms^2

Post Reply