More memory efficient handling of content
Posted: Sat Mar 16, 2013 7:20 pm
IMO it is relevant to discuss this matter in this section.
http://www.cmsimpleforum.com/viewtopic. ... 539#p34537
So basically the way cmsimple is designed, all the contents need to be accessible from a global variable. I think it's no good for use in a production environment if the site gets more popular. The content file is loaded and processed on each and every request. I find it wasteful to allocate memory for all of the site content. Only necessary content should be loaded.
I do understand if someone may find it a less of an issue but i don't think it's a good idea to read even several megabytes of data for each request on a limited shared hosting environment. It's hardly optimized.
EDIT: And from rfc() function in cms.php the content seems to get duplicated in another array while processing it (content[] copied in c[] -array in line cms.php:645), so if the content.htm was 5MB of size, it allocates more than 10MB. I have debugged it and it seems to do so. I'd call that a performance issue at least...
I thought of some ways to fix it and at the same time break the whole cms
There could be legacy mode- for all the old plugins that need to be updated so basically it'd populate the global variable with the content as usual but disabling it would process contents more efficiently. So this could be a kind of soft way to implement it.
EDIT3: pagemanager looks great, but is it necessary to have the plugin read the content file directly as it was already handled by the cms itself?
EDIT4: General API for plugins for easy access to the cms if needed
EDIT5: Contents (one page at a time) in a class object "model", and functions to retrieve content, modify it and search. Another model could be initialized with contents of another page when needed. Search results cache.
EDIT6: I have now made some quite succesful experiments. Viewing site content works fine, with only requested content populated in $c[] along with newsboxes. Reading content.htm partially works and does return the correct page information but editing is broken as it just dumps all content there each time when saved. So i will take a look at that. Pagemanager plugin is now broken like many others but the good thing is that i didn't manage to break the whole cms so there's hope for me to get the changes working. There is a content.map file which is serialized array containing header,link,startbyte,length,level information. I made all the handling in an external class. I don't know if the code is acceptable or not it may just become my very own fork of the cms.
http://www.cmsimpleforum.com/viewtopic. ... 539#p34537
So basically the way cmsimple is designed, all the contents need to be accessible from a global variable. I think it's no good for use in a production environment if the site gets more popular. The content file is loaded and processed on each and every request. I find it wasteful to allocate memory for all of the site content. Only necessary content should be loaded.
I do understand if someone may find it a less of an issue but i don't think it's a good idea to read even several megabytes of data for each request on a limited shared hosting environment. It's hardly optimized.
EDIT: And from rfc() function in cms.php the content seems to get duplicated in another array while processing it (content[] copied in c[] -array in line cms.php:645), so if the content.htm was 5MB of size, it allocates more than 10MB. I have debugged it and it seems to do so. I'd call that a performance issue at least...
I thought of some ways to fix it and at the same time break the whole cms
There could be legacy mode- for all the old plugins that need to be updated so basically it'd populate the global variable with the content as usual but disabling it would process contents more efficiently. So this could be a kind of soft way to implement it.
EDIT2: within download()- function it may be unnecessary to read and output a file with php as it of course reads them to the memory again... it could just redirect to the file.I have checked how it works. And if i'm not completely wrong the function in adm.php:728 shows how the content is handled on each request.
file: adm.php: line: 728
function read_content_file($path)
My suggestions:
- Generate "content map" with start- and end- bytes information, could be saved within content/pagedata.php???
- Content map would be regenerated each time the content is edited
- On each page load, only the necessary page content would be loaded.
offset= start reading at this byte
maxlen= end at this byte
file_get_contents($path . '/content/content.htm',false,null,offset,maxlen)
This would eliminate the need to read all the content within an array and find each pagebreak on every single page load. Huge difference!
- content.htm should be read in chunks only to make it possible to process bigger filesizes. Content.htm could contain the data split in chunks of 500 KB. One chunk could be read at a time when generating "the map" or searching. No separate files.
EDIT3: pagemanager looks great, but is it necessary to have the plugin read the content file directly as it was already handled by the cms itself?
EDIT4: General API for plugins for easy access to the cms if needed
EDIT5: Contents (one page at a time) in a class object "model", and functions to retrieve content, modify it and search. Another model could be initialized with contents of another page when needed. Search results cache.
EDIT6: I have now made some quite succesful experiments. Viewing site content works fine, with only requested content populated in $c[] along with newsboxes. Reading content.htm partially works and does return the correct page information but editing is broken as it just dumps all content there each time when saved. So i will take a look at that. Pagemanager plugin is now broken like many others but the good thing is that i didn't manage to break the whole cms so there's hope for me to get the changes working. There is a content.map file which is serialized array containing header,link,startbyte,length,level information. I made all the handling in an external class. I don't know if the code is acceptable or not it may just become my very own fork of the cms.