From abb6134095b3905e3178eaad00fbb705ba31c471 Mon Sep 17 00:00:00 2001 From: Bruce Momjian Date: Fri, 28 Sep 2001 18:30:05 +0000 Subject: [PATCH] Add to threads. --- doc/TODO.detail/thread | 694 ++++------------------------------------- 1 file changed, 60 insertions(+), 634 deletions(-) diff --git a/doc/TODO.detail/thread b/doc/TODO.detail/thread index 49af5600ea..c4d9eff98a 100644 --- a/doc/TODO.detail/thread +++ b/doc/TODO.detail/thread @@ -1,642 +1,68 @@ -From mscott@sacadia.com Wed Nov 15 14:50:19 2000 -Received: from goldengate.kojoworldwide.com. ([216.133.4.130]) - by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id OAA11583 - for ; Wed, 15 Nov 2000 14:50:13 -0500 (EST) -Received: from localhost (localhost [127.0.0.1]) - by goldengate.kojoworldwide.com. (8.9.1b+Sun/8.9.2) with ESMTP id LAA09998; - Wed, 15 Nov 2000 11:35:33 -0800 (PST) -Date: Wed, 15 Nov 2000 11:35:33 -0800 (PST) -From: Myron Scott -X-Sender: mscott@goldengate.kojoworldwide.com. -To: "Mikheev, Vadim" , - Bruce Momjian , Tom Lane -Subject: Please help with some advice -Message-ID: -MIME-Version: 1.0 -Content-Type: TEXT/PLAIN; charset=US-ASCII -Status: ORr - -Dear Sirs, - -I have been lurking on the PostgreSQL hackers list for about 3 months now -and your names comes up more than any with helpful info about the project -so I was hoping you could help me. - -Let me cut to the chase. I have been experimenting with 7.0.2 source to -see if I could create a mutlti-threaded version of the backend so -I could link directly from java ( I have a fe<->be protocol that I use for -my apps). Needless to say I got into much more than I bargained for. I -now have a version that works and it has some nice benefits that are very -helpful to a project that I am working on. What I gained was - -prepared statements outside of spi -batched commits (fsync) -one connection per thread - multiple threads per process - multiple processes per installation - -I never really intended for anyone else to see the work so I drifted -pretty far from the original code. I also ended up using Solaris threads -rather than pthreads, I did my own implementation of the bufmgr.c and -gram.y, and used Solaris implementation of mutex in place of S_LOCK and -TAS. I grabbed all global variables and put them in an environment -variable that is thread local. I also did some really stupid -things like making TransactionId uint64 and making all my inserts use the -same oid. - -My question is this. I would like to get some critical feedback and -suggestions about the work from others. What is the best way to go about -this? I thought about trying to create a project on greatbridge.org -but I am rather new to open source and the code needs commented properly -and cleaned up before too many try and look at it. - -Any suggestions would be greatly appreciated. - - -Thanks in advance, - -Myron Scott - - - -From mscott@sacadia.com Thu Nov 16 17:19:45 2000 -Received: from goldengate.kojoworldwide.com. ([216.133.4.130]) - by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id RAA04315 - for ; Thu, 16 Nov 2000 17:19:43 -0500 (EST) -Received: from localhost (localhost [127.0.0.1]) - by goldengate.kojoworldwide.com. (8.9.1b+Sun/8.9.2) with ESMTP id OAA11449; - Thu, 16 Nov 2000 14:05:15 -0800 (PST) -Date: Thu, 16 Nov 2000 14:05:15 -0800 (PST) -From: Myron Scott -X-Sender: mscott@goldengate.kojoworldwide.com. -To: Bruce Momjian -cc: "Mikheev, Vadim" , Tom Lane -Subject: Re: Please help with some advice -In-Reply-To: <200011160533.AAA27886@candle.pha.pa.us> -Message-ID: -MIME-Version: 1.0 -Content-Type: TEXT/PLAIN; charset=US-ASCII -Status: OR - -Bruce Momjian wrote: - ->I am curious how you isolated each thread. It seems we pretty much ->assume all our memory is controlled by a single query in the process. - - -I moved all global variables to a thread global variable which is accessed -by the method GetEnv(). Which looks like this - -Env* GetEnv(void) { - Env* env; - thr_getspecific(*envkey,(void*)&env); - return env; -} - -The Env struct includes the CurrentMemoryContext, TopMemoryContext, -PortalHeapMemory for each instance of a connection (one thread per -connection). So, for example, -EndPortalAllocMode uses GetEnv()->CurrentMemoryContext - -void -EndPortalAllocMode() -{ - PortalHeapMemory context; - - AssertState(PortalManagerEnabled); - AssertState(IsA(GetEnv()->CurrentMemoryContext, -PortalHeapMemory)); - - context = (PortalHeapMemory) GetEnv()->CurrentMemoryContext; - AssertState(PointerIsValid(context->block)); /* XXX -Trap(...) */ - - /* free current mode */ - AllocSetReset(&HEAPMEMBLOCK(context)->setData); - MemoryContextFree((MemoryContext) -PortalHeapMemoryGetVariableMemory(context), - context->block); - - /* restore previous mode */ - context->block = FixedStackPop(&context->stackData); -} - - - - -From vmikheev@SECTORBASE.COM Thu Nov 16 17:23:22 2000 -Received: from sectorbase2.sectorbase.com ([208.48.122.131]) - by candle.pha.pa.us (8.9.0/8.9.0) with SMTP id RAA04562 - for ; Thu, 16 Nov 2000 17:23:21 -0500 (EST) -Received: by sectorbase2.sectorbase.com with Internet Mail Service (5.5.2650.21) - id ; Thu, 16 Nov 2000 14:05:24 -0800 -Message-ID: <8F4C99C66D04D4118F580090272A7A234D318D@sectorbase1.sectorbase.com> -From: "Mikheev, Vadim" -To: "'Myron Scott'" , - Bruce Momjian - -Cc: Tom Lane -Subject: RE: Please help with some advice -Date: Thu, 16 Nov 2000 14:09:30 -0800 -MIME-Version: 1.0 -X-Mailer: Internet Mail Service (5.5.2650.21) -Content-Type: text/plain; - charset="iso-8859-1" -Status: ORr - -I think the question do we want to make backend multy-threaded -should be discussed in hackers. - -Vadim - -> -----Original Message----- -> From: Myron Scott [mailto:mscott@sacadia.com] -> Sent: Thursday, November 16, 2000 2:05 PM -> To: Bruce Momjian -> Cc: Mikheev, Vadim; Tom Lane -> Subject: Re: Please help with some advice -> -> -> Bruce Momjian wrote: -> -> >I am curious how you isolated each thread. It seems we pretty much -> >assume all our memory is controlled by a single query in the process. -> -> -> -> I moved all global variables to a thread global variable -> which is accessed -> by the method GetEnv(). Which looks like this -> -> Env* GetEnv(void) { -> Env* env; -> thr_getspecific(*envkey,(void*)&env); -> return env; -> } -> -> The Env struct includes the CurrentMemoryContext, TopMemoryContext, -> PortalHeapMemory for each instance of a connection (one thread per -> connection). So, for example, -> EndPortalAllocMode uses GetEnv()->CurrentMemoryContext -> -> void -> EndPortalAllocMode() -> { -> PortalHeapMemory context; -> -> AssertState(PortalManagerEnabled); -> AssertState(IsA(GetEnv()->CurrentMemoryContext, -> PortalHeapMemory)); -> -> context = (PortalHeapMemory) GetEnv()->CurrentMemoryContext; -> AssertState(PointerIsValid(context->block)); /* XXX -> Trap(...) */ -> -> /* free current mode */ -> AllocSetReset(&HEAPMEMBLOCK(context)->setData); -> MemoryContextFree((MemoryContext) -> PortalHeapMemoryGetVariableMemory(context), -> context->block); -> -> /* restore previous mode */ -> context->block = FixedStackPop(&context->stackData); -> } -> -> -> - -From mscott@sacadia.com Thu Nov 16 22:16:38 2000 -Received: from goldengate.kojoworldwide.com. ([216.133.4.130]) - by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id WAA14638 - for ; Thu, 16 Nov 2000 22:16:36 -0500 (EST) -Received: from localhost (localhost [127.0.0.1]) - by goldengate.kojoworldwide.com. (8.9.1b+Sun/8.9.2) with ESMTP id TAA11874; - Thu, 16 Nov 2000 19:04:48 -0800 (PST) -Date: Thu, 16 Nov 2000 19:04:48 -0800 (PST) -From: Myron Scott -X-Sender: mscott@goldengate.kojoworldwide.com. -To: Bruce Momjian -cc: "Mikheev, Vadim" , Tom Lane -Subject: Re: Please help with some advice -In-Reply-To: <200011170156.UAA11438@candle.pha.pa.us> -Message-ID: -MIME-Version: 1.0 -Content-Type: TEXT/PLAIN; charset=US-ASCII -Status: ORr - -Thanks very much, I will post to hackers. - -Myron - - - -From pgsql-hackers-owner+M2691@postgresql.org Tue Jan 2 00:30:20 2001 -Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28]) - by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id AAA08195 - for ; Tue, 2 Jan 2001 00:30:19 -0500 (EST) -Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28]) - by mail.postgresql.org (8.11.1/8.11.1) with SMTP id f025UjL33335; - Tue, 2 Jan 2001 00:30:45 -0500 (EST) - (envelope-from pgsql-hackers-owner+M2691@postgresql.org) -Received: from mailsys01.intnet.net (tmail.wwc.com [198.252.32.143] (may be forged)) - by mail.postgresql.org (8.11.1/8.11.1) with ESMTP id f025UTL33232 - for ; Tue, 2 Jan 2001 00:30:32 -0500 (EST) - (envelope-from mscott@sacadia.com) -Received: from [206.112.108.0] (HELO sacadia.com) - by mailsys01.intnet.net (CommuniGate Pro SMTP 3.3.2) - with ESMTP id 2214231; Tue, 02 Jan 2001 00:29:47 -0500 -Message-ID: <3A5167DB.3050807@sacadia.com> -Date: Mon, 01 Jan 2001 21:32:11 -0800 -From: Myron Scott -Reply-To: mscott@sacadia.com -User-Agent: Mozilla/5.0 (Windows; U; Win98; en-US; m18) Gecko/20001108 Netscape6/6.0 +From pgsql-hackers-owner+M13599=candle.pha.pa.us=pgman@postgresql.org Wed Sep 26 17:25:32 2001 +Return-path: +Received: from server1.pgsql.org (server1.pgsql.org [64.39.15.238] (may be forged)) + by candle.pha.pa.us (8.11.6/8.10.1) with ESMTP id f8QLPWo07589 + for ; Wed, 26 Sep 2001 17:25:32 -0400 (EDT) +Received: from postgresql.org (webmail.postgresql.org [216.126.85.28]) + by server1.pgsql.org (8.11.6/8.11.6) with ESMTP id f8QLPf405606 + for ; Wed, 26 Sep 2001 16:25:41 -0500 (CDT) + (envelope-from pgsql-hackers-owner+M13599=candle.pha.pa.us=pgman@postgresql.org) +Received: from gromit.dotclick.com (ipn9-f8366.net-resource.net [216.204.83.66]) + by postgresql.org (8.11.3/8.11.4) with ESMTP id f8QKj3h82020 + for ; Wed, 26 Sep 2001 16:45:03 -0400 (EDT) + (envelope-from markw@mohawksoft.com) +Received: from mohawksoft.com (IDENT:markw@localhost.localdomain [127.0.0.1]) + by gromit.dotclick.com (8.9.3/8.9.3) with ESMTP id QAA23693; + Wed, 26 Sep 2001 16:43:02 -0400 +Message-ID: <3BB23DD6.E86AF327@mohawksoft.com> +Date: Wed, 26 Sep 2001 16:43:02 -0400 +From: mlw +X-Mailer: Mozilla 4.75 [en] (X11; U; Linux 2.4.2 i686) X-Accept-Language: en MIME-Version: 1.0 -To: "Ross J. Reedstrom" -CC: pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] Using Threads? -References: <004401c058fd$fd498d40$f2356880@tracy> <20001204113307.B5871@rice.edu> -Content-Type: text/plain; charset=us-ascii; format=flowed -Content-Transfer-Encoding: 7bit -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - -For anyone interested, - -I have posted my multi-threaded version of PostgreSQL here. - -http://www.sacadia.com/mtpg.html - -It is based on 7.0.2 and the TAO CORBA ORB which is here. - -http://www.cs.wustl.edu/~schmidt/TAO.html - -Myron Scott -mkscott@sacadia.com - - - -From bright@fw.wintelcom.net Tue Jan 2 03:02:28 2001 -Received: from fw.wintelcom.net (bright@ns1.wintelcom.net [209.1.153.20]) - by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id DAA16169 - for ; Tue, 2 Jan 2001 03:02:27 -0500 (EST) -Received: (from bright@localhost) - by fw.wintelcom.net (8.10.0/8.10.0) id f0282Vm10623; - Tue, 2 Jan 2001 00:02:31 -0800 (PST) -Date: Tue, 2 Jan 2001 00:02:31 -0800 -From: Alfred Perlstein -To: Bruce Momjian -Cc: Tom Lane , pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] Assuming that TAS() will succeed the first time is verboten -Message-ID: <20010102000230.C19572@fw.wintelcom.net> -References: <9850.978067943@sss.pgh.pa.us> <200101020759.CAA15836@candle.pha.pa.us> -Mime-Version: 1.0 +To: "D. Hageman" , + "pgsql-hackers@postgresql.org" +Subject: Re: [HACKERS] Spinlock performance improvement proposal +References: Content-Type: text/plain; charset=us-ascii -Content-Disposition: inline -User-Agent: Mutt/1.2.5i -In-Reply-To: <200101020759.CAA15836@candle.pha.pa.us>; from pgman@candle.pha.pa.us on Tue, Jan 02, 2001 at 02:59:20AM -0500 -Status: OR - -* Bruce Momjian [010101 23:59] wrote: -> > Alfred Perlstein writes: -> > > One trick that may help is calling sched_yield(2) on a lock miss, -> > > it's a POSIX call and quite new so you'd need a 'configure' test -> > > for it. -> > -> > The author of the current s_lock code seems to have thought that -> > select() with a zero delay would do the equivalent of sched_yield(). -> > I'm not sure if that's true on very many kernels, if indeed any... -> > -> > I doubt we could buy much by depending on sched_yield(); if you want -> > to assume POSIX facilities, ISTM you might as well go for user-space -> > semaphores and forget the whole TAS mechanism. -> -> -> Another issue is that sched_yield brings in the pthreads library/hooks -> on some OS's, which we certainly want to avoid. - -I know it's a major undertaking, but since the work is sort of done, -have you guys considered the port to solaris threads and seeing about -making a pthreads port of that? - -I know it would probably get you considerable gains under Windows -at the expense of dropping some really really legacy system. - -Or you could do what apache (is rumored) does and have it do either -threads or processes or both... - --- --Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org] -"I have the heart of a child; I keep it in a jar on my desk." - -From pgsql-hackers-owner+M4275@postgresql.org Mon Feb 5 21:45:00 2001 -Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28]) - by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id VAA09262 - for ; Mon, 5 Feb 2001 21:44:59 -0500 (EST) -Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28]) - by mail.postgresql.org (8.11.1/8.11.1) with SMTP id f162ixx00920; - Mon, 5 Feb 2001 21:44:59 -0500 (EST) - (envelope-from pgsql-hackers-owner+M4275@postgresql.org) -Received: from goldengate.kojoworldwide.com. ([216.133.4.130]) - by mail.postgresql.org (8.11.1/8.11.1) with ESMTP id f162fSx00595 - for ; Mon, 5 Feb 2001 21:41:29 -0500 (EST) - (envelope-from mscott@sacadia.com) -Received: from localhost (localhost [127.0.0.1]) - by goldengate.kojoworldwide.com. (8.9.1b+Sun/8.9.2) with ESMTP id SAA03298 - for ; Mon, 5 Feb 2001 18:25:05 -0800 (PST) -Date: Mon, 5 Feb 2001 18:25:05 -0800 (PST) -From: Myron Scott -X-Sender: mscott@goldengate.kojoworldwide.com. -To: pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] Using Threads? -Message-ID: -MIME-Version: 1.0 -Content-Type: TEXT/PLAIN; charset=US-ASCII -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - -I have put a new version of my multi-threaded -postgresql experiment at - -http://www.sacadia.com/mtpg.html - -This one actually works. I have added a server -based on omniORB, a CORBA 2.3 ORB from ATT. It - is much smaller than TAO and uses the thread per -connection model. I haven't added the java side -of the JNI interface yet but the C++ side is there. - -It's still not stable but it is much better than -the last. - -Myron Scott -mkscott@sacadia.com - - - - - -From pgsql-hackers-owner+M4304@postgresql.org Tue Feb 6 10:24:21 2001 -Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28]) - by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id KAA22027 - for ; Tue, 6 Feb 2001 10:24:20 -0500 (EST) -Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28]) - by mail.postgresql.org (8.11.1/8.11.1) with SMTP id f16FOBx97182; - Tue, 6 Feb 2001 10:24:11 -0500 (EST) - (envelope-from pgsql-hackers-owner+M4304@postgresql.org) -Received: from goldengate.kojoworldwide.com. ([216.133.4.130]) - by mail.postgresql.org (8.11.1/8.11.1) with ESMTP id f16FLWx96814 - for ; Tue, 6 Feb 2001 10:21:33 -0500 (EST) - (envelope-from mscott@sacadia.com) -Received: from localhost (localhost [127.0.0.1]) - by goldengate.kojoworldwide.com. (8.9.1b+Sun/8.9.2) with ESMTP id HAA04170; - Tue, 6 Feb 2001 07:05:04 -0800 (PST) -Date: Tue, 6 Feb 2001 07:05:04 -0800 (PST) -From: Myron Scott -X-Sender: mscott@goldengate.kojoworldwide.com. -To: Karel Zak -cc: pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] Using Threads -In-Reply-To: -Message-ID: -MIME-Version: 1.0 -Content-Type: TEXT/PLAIN; charset=US-ASCII -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - - -> -> Sorry I haven't time to see and test your experiment, -> but I have a question. How you solve memory management? -> The current mmgr is based on global variable -> CurrentMemoryContext that is very often changed and used. -> Use you for this locks? If yes it is probably problematic -> point for perfomance. -> -> Karel -> - -There are many many globals I had to work around including all the memory -management stuff. I basically threw everything into and "environment" -variable which I stored in a thread specific using thr_setspecific. - -Performance is acually very good for what I am doing. I was able to batch -commit transactions which cuts down on fsync calls, use prepared -statements from my client using CORBA, and the various locking calls for -the threads (cond_wait,mutex_lock, and sema_wait) seem pretty fast. I did -some performance tests for inserts - -20 clients, 900 inserts per client, 1 insert per transaction, 4 different -tables. - -7.0.2 About 10:52 average completion -multi-threaded 2:42 average completion -7.1beta3 1:13 average completion - -If I increased the number of inserts per transaction, multi-threaded got -closer to 7.1 for inserts. I haven't tested other other types of -commands -yet. - - -Myron Scott -mkscott@sacadia.com - - -From pgsql-hackers-owner+M4313@postgresql.org Tue Feb 6 12:32:00 2001 -Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28]) - by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id MAA29163 - for ; Tue, 6 Feb 2001 12:31:59 -0500 (EST) -Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28]) - by mail.postgresql.org (8.11.1/8.11.1) with SMTP id f16HVox17454; - Tue, 6 Feb 2001 12:31:51 -0500 (EST) - (envelope-from pgsql-hackers-owner+M4313@postgresql.org) -Received: from ara.zf.jcu.cz (ara.zf.jcu.cz [160.217.161.4]) - by mail.postgresql.org (8.11.1/8.11.1) with ESMTP id f16HV6x17323 - for ; Tue, 6 Feb 2001 12:31:06 -0500 (EST) - (envelope-from zakkr@zf.jcu.cz) -Received: from localhost (zakkr@localhost) - by ara.zf.jcu.cz (8.9.3/8.9.3/Debian 8.9.3-21) with SMTP id SAA03980; - Tue, 6 Feb 2001 18:31:02 +0100 -Date: Tue, 6 Feb 2001 18:31:02 +0100 (CET) -From: Karel Zak -To: Myron Scott -cc: pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] Using Threads -In-Reply-To: -Message-ID: -MIME-Version: 1.0 -Content-Type: TEXT/PLAIN; charset=US-ASCII -Precedence: bulk -Sender: pgsql-hackers-owner@postgresql.org -Status: OR - - -On Tue, 6 Feb 2001, Myron Scott wrote: - -> There are many many globals I had to work around including all the memory -> management stuff. I basically threw everything into and "environment" -> variable which I stored in a thread specific using thr_setspecific. - - Yes, it's good. I working on multi-thread application server -(http://mape.jcu.cz) and I use for this project some things from PG (like -mmgr), I planning use same solution. - -> Performance is acually very good for what I am doing. I was able to batch -> commit transactions which cuts down on fsync calls, use prepared -> statements from my client using CORBA, and the various locking calls for -> the threads (cond_wait,mutex_lock, and sema_wait) seem pretty fast. I did -> some performance tests for inserts -> -> 20 clients, 900 inserts per client, 1 insert per transaction, 4 different -> tables. -> -> 7.0.2 About 10:52 average completion -> multi-threaded 2:42 average completion -> 7.1beta3 1:13 average completion - -It is very very good for time for 7.1, already look forward to 7.2! :-) - - BTW, I not sure if you anytime in future will see threads in -official PostgreSQL and if you spending time on relevant things (IMHO). - - Karel - - - - - - -From pgsql-hackers-owner+M4304@postgresql.org Tue Feb 6 10:24:21 2001 -Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28]) - by candle.pha.pa.us (8.9.0/8.9.0) with ESMTP id KAA22027 - for ; Tue, 6 Feb 2001 10:24:20 -0500 (EST) -Received: from mail.postgresql.org (webmail.postgresql.org [216.126.85.28]) - by mail.postgresql.org (8.11.1/8.11.1) with SMTP id f16FOBx97182; - Tue, 6 Feb 2001 10:24:11 -0500 (EST) - (envelope-from pgsql-hackers-owner+M4304@postgresql.org) -Received: from goldengate.kojoworldwide.com. ([216.133.4.130]) - by mail.postgresql.org (8.11.1/8.11.1) with ESMTP id f16FLWx96814 - for ; Tue, 6 Feb 2001 10:21:33 -0500 (EST) - (envelope-from mscott@sacadia.com) -Received: from localhost (localhost [127.0.0.1]) - by goldengate.kojoworldwide.com. (8.9.1b+Sun/8.9.2) with ESMTP id HAA04170; - Tue, 6 Feb 2001 07:05:04 -0800 (PST) -Date: Tue, 6 Feb 2001 07:05:04 -0800 (PST) -From: Myron Scott -X-Sender: mscott@goldengate.kojoworldwide.com. -To: Karel Zak -cc: pgsql-hackers@postgresql.org -Subject: Re: [HACKERS] Using Threads -In-Reply-To: -Message-ID: -MIME-Version: 1.0 -Content-Type: TEXT/PLAIN; charset=US-ASCII +Content-Transfer-Encoding: 7bit Precedence: bulk Sender: pgsql-hackers-owner@postgresql.org Status: OR - -> -> Sorry I haven't time to see and test your experiment, -> but I have a question. How you solve memory management? -> The current mmgr is based on global variable -> CurrentMemoryContext that is very often changed and used. -> Use you for this locks? If yes it is probably problematic -> point for perfomance. -> -> Karel -> - -There are many many globals I had to work around including all the memory -management stuff. I basically threw everything into and "environment" -variable which I stored in a thread specific using thr_setspecific. - -Performance is acually very good for what I am doing. I was able to batch -commit transactions which cuts down on fsync calls, use prepared -statements from my client using CORBA, and the various locking calls for -the threads (cond_wait,mutex_lock, and sema_wait) seem pretty fast. I did -some performance tests for inserts - -20 clients, 900 inserts per client, 1 insert per transaction, 4 different -tables. - -7.0.2 About 10:52 average completion -multi-threaded 2:42 average completion -7.1beta3 1:13 average completion - -If I increased the number of inserts per transaction, multi-threaded got -closer to 7.1 for inserts. I haven't tested other other types of -commands -yet. - - -Myron Scott -mkscott@sacadia.com - - -From lamar.owen@wgcr.org Thu Jun 28 11:14:10 2001 -Return-path: -Received: from www.wgcr.org (IDENT:root@www.wgcr.org [206.74.232.194]) - by candle.pha.pa.us (8.10.1/8.10.1) with ESMTP id f5SFE9U18758 - for ; Thu, 28 Jun 2001 11:14:09 -0400 (EDT) -Received: from lowen.wgcr.org (IDENT:lowen@[10.1.2.3]) - by www.wgcr.org (8.9.3/8.9.3/WGCR) with SMTP id LAA11879; - Thu, 28 Jun 2001 11:14:14 -0400 -Content-Type: text/plain; - charset="iso-8859-1" -From: Lamar Owen -To: Bruce Momjian -Subject: Process weight (was:Re: [GENERAL] Re: Red Hat to support PostgreSQL) -Date: Thu, 28 Jun 2001 11:14:09 -0400 -X-Mailer: KMail [version 1.2] -References: <200106272258.f5RMwIb26959@candle.pha.pa.us> -In-Reply-To: <200106272258.f5RMwIb26959@candle.pha.pa.us> -MIME-Version: 1.0 -Message-ID: <01062811140902.01118@lowen.wgcr.org> -Content-Transfer-Encoding: 8bit -Status: ORr - -On Wednesday 27 June 2001 18:58, Bruce Momjian wrote: -> > I had almost given up on using Postgres for this system because under -> > Solaris, it just couldn't cut it (MySQL could do the work with one CPU -> > while Postgres took up even more CPU and required *both* CPUs to be -> > enabled), but when we moved the system to a Linux box, things worked -> > much better. - -> Ah, back to a PostgreSQL topic. :-) - -> My guess on this one is that Solaris is slower for PostgreSQL because -> process switching is _much_ heavier on Solaris than other OS's. This is -> because of the way they implemented processes in SVr4. They got quite -> heavy, almost requiring kernel threads so you weren't switching -> processes all the time. - -Now, the question of the week: -Is supporting a thread model for an inefficient OS a desirable thing to do, -when more efficient OS kernels are available such as FreeBSD 4.x and Linux -2.4? My opinion is that our existing model, when used with a -connection-pooling frontend, is rather efficient. (Yes, I use a -connection-pooling frontend. Performance is rather nice, and I don't have to -have a full backend spawned for every page hit.) - -In fact, on a Linux box threads show as processes. While I know that the -kernel actually supports themin a slightly different manner than processes, -they have more similarities than differences. - -However, even on OS's where threads are supported, the mechanism to support -those threads must be an efficient one -- not all pthreads libraries are -created equal. Many are frontends (expensive ones, at that) for plain old -processes. - -Does anyone know of a resource that details the 'weight' of processes for our -supported platforms? [reply off-list -- I'll be glad to summarize responses -to HACKERS, ADMIN, or PORTS, as appropriate, if desired.] --- -Lamar Owen -WGCR Internet Radio -1 Peter 4:11 +"D. Hageman" wrote: + +> The plan for the new spinlocks does look like it has some potential. My +> only comment in regards to permformance when we start looking at SMP +> machines is ... it is my belief that getting a true threaded backend may +> be the only way to get the full potential out of SMP machines. I see that +> is one of the things to experiment with on the TODO list and I have seen +> some people have messed around already with this using Solaris threads. +> It should probably be attempted with pthreads if PostgreSQL is going to +> keep some resemblance of cross-platform compatibility. At that time, it +> would probably be easier to go in and clean up some stuff for the +> implementation of other TODO items (put in the base framework for more +> complex future items) as threading the backend would take a little bit of +> ideology shift. + +I can only think of two objectives for threading. (1) running the various +connections in their own thread instead of their own process. (2) running +complex queries across multiple threads. + +For item (1) I see no value to this. It is a lot of work with no tangible +benefit. If you have an old fashion pthreads implementation, it will hurt +performance because are scheduled within the single process's time slice.. If +you have a newer kernel scheduled implementation, then you will have the same +scheduling as separate processes. The only thing you will need to do is +switch your brain from figuring out how to share data, to trying to figure +out how to isolate data. A multithreaded implementation lacks many of the +benefits and robustness of a multiprocess implementation. + +For item (2) I can see how that could speed up queries in a low utilization +system, and that would be cool, but in a server that is under load, threading +the queries probably be less efficient. + + +---------------------------(end of broadcast)--------------------------- +TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org -- 2.40.0