Analyzing Data

[Xrproto]

Alright so I found some source code in Java that was a nice little data analyzer for keywords that would give me info like top 100 keywords in my database and so on. But I've already broke it.

Here's what I'm doing, I have a database that I dump search queries into so I can analyze them later one. I searched for a pre-built option before trying to build my own when I found said app mentioned above. Worked great up to about 1million query results then I turned my my script to get more data for a couple of hours.

Now that I've hit 2.5+ million results in my database that little java app can't handle it and crashes. I don't know how to code in Java and didn't plan on learning anytime soon. So I'm open for suggestions on how I should approach this.

Some thing to note it is that with in a day I'm projecting anywhere between 8-10 million results in the database I would like to analyze once I turn everything on after testing.

It doesn't matter the language if you have one to suggest I'm open.

 

[Bofu2U]

Maybe it's too many records for the java app, try adding a LIMIT on the end of the query?

ex: LIMIT 0,500000
that will read the first 500,000 records..

Not sure if you can even edit the query with the script, but just trying to help.

 

[Xrproto]

Maybe it's too many records for the java app, try adding a LIMIT on the end of the query?

ex: LIMIT 0,500000
that will read the first 500,000 records..

Not sure if you can even edit the query with the script, but just trying to help.

Yeah I can do that, I cut it off at 1million and it worked. But I want to be able to analyze all the data and I don't want to go down a path of code that will end up leaving me limited in the end.

I'm okay with PHP and it's been several years since I coded in C++ so I might even code something in C++.

 

[Bofu2U]

Why not just do the first query for 1 mil, and the 2nd query for another 1 mil? Won't it still work with multiple queries?

PS - If you find a good pre-written PHP version let me know! I'm looking to modify one for my own needs

 

[Xrproto]

Why not just do the first query for 1 mil, and the 2nd query for another 1 mil? Won't it still work with multiple queries?

PS - If you find a good pre-written PHP version let me know! I'm looking to modify one for my own needs

I will do something like that for certain times of the day but I need to come up with something that can handle a whole day worth of results.

If I come up with a php version I will let you know.

 

[Xrproto]

Does anyone know of anything like Oracle Discoverer but for mysql?

I know how to use discoverer and could get what I want with that if they have something like that for mysql.

 

[infoTim]

Now that I've hit 2.5+ million results in my database that little java app can't handle it and crashes. I don't know how to code in Java and didn't plan on learning anytime soon. So I'm open for suggestions on how I should approach this.

Why is it crashing...is there any info printed? You might just be able to start it with more memory allocated to the JVM on the command line and be ok.

 

[Xrproto]

Why is it crashing...is there any info printed? You might just be able to start it with more memory allocated to the JVM on the command line and be ok.

No info printed, actually when first run it's suppose to save all the info in a file then I load it to be viewed in charts and tables.

I'm launching it in eclipse SDK, I never coded in java so I don't know my way around it yet. But after you said that I ran it again and I get an outofmemoryerror, I never thought to look in the console to see what it was doing.

Now how do I fix that?

 

[Xrproto]

I've found out what I need to do and that's change the heap size, I should be able to figure that out. Hopefully it works