forum with zero sql

threadmark

In Runtime
Messages
134
Location
somewhere in this general direction.
Using a RAM disk with a powerful bit of code to format html pages would require zero sql. My Perl CGI ports the data to C in order to format the pages. From C I have the formatted code added to the required section using code “thread marking” : ) . A technique, im sure has a proper name but I called it this, it formats a hex string that is unique to that particular post and formats that hex string into a hidden variable on the html page tagged to the post. When a user wants to edit or moderate the post, the perl cgi passes the variables to C in which it can determine what part of the html page to format.
That way when people are searching/browsing the forum, it would only require downloads and no sql. The only process hungry occurrence is in the cgi only…yeah baby!!

I overcome the volatile memory with had links to backup.
 
Why would you want to go around your ankle to scratch your nose?
A sql database makes the search for anything in a forum a lot simpler. As for editing you've made it too complex on the back end.
What happens when you have a hiccup in the system like ram disk failure?
 
Why would you want to go around your ankle to scratch your nose?
A sql database makes the search for anything in a forum a lot simpler. As for editing you've made it too complex on the back end.
What happens when you have a hiccup in the system like ram disk failure?

that's silly, ram failure would do the same to SQL. All I have to do is install a new one then reboot. so what's your point. As far as explaining WHY I'm doing it this way, is to use as little process power as possible. For general forum searching/browsing would not take much if any process power at all.

I could run a forum off a computer the size of a credit card :)
 
You'd think that.

I did a simillar thing using a bash script and various utilities to list a directory and search on the fly and present pages with HTML on my home media centre machine, (a 200Mhz MIPS machine) it's probably more resource intensive trying to make your own flat file database than it is to use an SQL database.

I'm not sure I get your point about RAM failure in SQL, sure a ram failure will obliterate any existing clustered index, BUT, with a set of good maintenance plans and a fair idea of what you're doing your indexes shouldn't be too hard to rebuild.
PLUS, for a large forum, not every index/data is going to be stored in RAM, because well that's silly, especially so for returning results from searches, the workflow can be optimised, but this is always going to be a disk intensive activity as it'd be impossible to ram cache all the data you wanted to.

this idea of using a RAM disk... what's the point? if you wanted a seriously fast database you'd just use solid state disks SAS disks, connected by fibre to your DB servers...

and you'd use a proper database that was designed with what you want to do in mind, (depends on the sort of data you;re searching/returning.)

suffice to say I may have missed the point of what you;re trying to do, but I'm reasonably sure that what you're trying to do is a bit weird, and can be better done with a properly designed and optimised database server that's accessible using SQL
 
You'd think that.

I did a simillar thing using a bash script and various utilities to list a directory and search on the fly and present pages with HTML on my home media centre machine, (a 200Mhz MIPS machine) it's probably more resource intensive trying to make your own flat file database than it is to use an SQL database.

I'm not sure I get your point about RAM failure in SQL, sure a ram failure will obliterate any existing clustered index, BUT, with a set of good maintenance plans and a fair idea of what you're doing your indexes shouldn't be too hard to rebuild.
PLUS, for a large forum, not every index/data is going to be stored in RAM, because well that's silly, especially so for returning results from searches, the workflow can be optimised, but this is always going to be a disk intensive activity as it'd be impossible to ram cache all the data you wanted to.

this idea of using a RAM disk... what's the point? if you wanted a seriously fast database you'd just use solid state disks SAS disks, connected by fibre to your DB servers...

and you'd use a proper database that was designed with what you want to do in mind, (depends on the sort of data you;re searching/returning.)

suffice to say I may have missed the point of what you;re trying to do, but I'm reasonably sure that what you're trying to do is a bit weird, and can be better done with a properly designed and optimised database server that's accessible using SQL

I don't think you read my post properly.
I'm using a hard link to mirror the data on a redundant drive. Running on a ramdisk is to compensate and is not crucial. The engineering feet is the fact it is saving process power. In this forum you use PHP and in order to display information like posts, the code needs to run a SQL query. My method does not.
 
You're not saving processing power.

That's the point.
from experiance, saving things as whole pages is slow. it means that when you want to search for something you have to open a whole document and parse that document. you don't have a database so you always need to parse things (including lists of files) in memory.

I've long since put away my media player, but do you have any idea what the code is like to get a directory listing, split that into strings, store in an array, then look at file types, format to HTML put into a visual tree structure, and display different icons depending on file type? unless you're writing this with it storing to temporary files as well you need to do this all in one line.

many may years ago, I actually did exactly what you're doing, with a program in C that put an untypable character into a text file, that works well as a separator as you can use getc to load characters from the file and decide to stop when you get to the special marker char that you can use to mark column, and a char that you can use to mark ends of rows.


what you're not understanding is that, it's trivially easy to create a flat file database, but SQL isn't a flat file database, it's a relational database.

so to retrieve this post, the page isn't just loaded.

first the title page is loaded, where results are sorted by date, (though you can change the sorting if you want?) then you click on the link, where posts can be sorted by date or by reply, and can be ascending or descending, then post content is linked to the member details by use of the members unique ID.

which means that member information, (username, location post count title etc) can change easily, and be updated with every post...

the point is, I understand what you're doing, what you're doing increases RAM usage, particularly because when you get larger files they end up needing to be opened in memory, and your file stream ends up in memory, to combat this you intend to use a RAM disk? I assume that you mean a swap or page file as oppose to a ram disk/hyperdrive (HyperOs OneClick)

as for reduced processor...
I'm not convinced about that either, it'd be interesting to see how your file based search system compares against a modern day SQL system, also interesting to see how your system compares against some older version of Oracle, or SQL server...

I don't doubt your ability to run on a credit card sized PC given the size of the raspberry pi, and it's 700MHz processor and 500 MB RAM, but how would that stack up against a system running MS SQL 6.5. that had amazing performance on 66MHz systems...


I understand what you're doing, but you're throwing a lot of expensive disk resource at slow code, that is trying to reduce processor usage where a CPU problem just doesn't exist?

Sure you'll never run MS SQL2012 on a credit card sized PC, (actually you probably could) -and you'd probably get results loaded faster than what you're doing.
but a decent light weight database server can happily run on limited resources and return results very fast.
 
OK, I'll try and eliminate some of your confusion.
In a massive forum you would receive about 100-200 posted per second from users. That's the only time my server would be strained. Only when a user POST's, is my server under the pump. Just to give you an idea, i have performed a test using over 5000 computers to artificially test the server. I have tested using a Pentium 3 and it can handle 1000 at once. I don't understand why you can't see the benefits. Granted I have only tested artificially but the results have poked me to develop it further.
 
Let me get this straight.
You rebuild the page in its entirety every time.
Posting is what loads up your server.
You buffer the data in a ramdisk.
You have no search engine other than a unique identifier embedded in each post.
Your code has to do a 2 way conversion. Posting, reading.
You have no catastrophic failure back up plan.
HTML, PHP, XML, and CSS with a SQL database are far more efficient. Cache-ing pages reduces page load times. And decreases server loads as the page doesn't have to be completely rebuilt every time it's look at or posted to. Just those parts that need updated.
What are you doing for appearance? How are the pages laid out? Where are you storing your page graphics? How about a screen shot?
 
I don't see the benefits because of all the excess IO ops the processor will have to do to open the data in the memory, send it (or save it) then close the the "file" The opening and closing is time consuming, and more intensive than a well-laid out database that has indexing done properly. This isn't even considering that you'd then need to format the results into some format that you can use it, where the database and drive you use to tie it to your code will do this for you with little effort.

How is that "only when a user POST's, is my server under the pump" true? when they load the page, wouldn't it draw from the server so they can see posts? When google spiders run across it wouldn't their loads have to pull from the server?

Also, what about concurrency? Two people make a post in a thread at literally the exact same time - RAM (and thus the RAMDISK) isn't so good at dealing with this. One value will be lost. Databases on the other hand will see both posts and lock one out until the other finishes so they both get written to the system.
 
I pretty much agree with the replies here. While what you might seem to have is a super-fast, innovative way of dealing with the data, what you've actually done is re-invented a wheel that's likely no faster than using existing methods set up properly, and is likely ridiculously fragile in the event of a system / power failure (yes, it's mirrored to disk, but how often? Are you using some foolproof checking algorithm or just syncing them up every so often? If it's the latter then this isn't resilient at all.)

If someone did need something akin to what you're doing, then the normal thing to do would be to setup MySQL / PostgreSQL and then if that was too slow, allocate part of it to a RAMDisk to speed up the queries. The point is that MySQL queries are rarely some big bottleneck in a system (unless they're hideously complicated or badly written), and where they are an issue there's many things that would alleviate this in a stable, industry standard way, that probably can get things faster and more efficient than your method with the right configuration.
 
Back
Top Bottom