509020 - Make tamarin play nice with valgrind

Status: RESOLVED WONTFIX

(Tamarin Graveyard :: Garbage Collection (mmGC), defect)

Description

[Edwin Smith]

At least:

- need a suppressions file for MMgc since the conservative mark code apparently reads un-initialized memory.  on the stack, at least, possibly more.

- enable MMGC_USE_SYSTEM_MALLOC so we get valgrind inspection of new/delete/malloc/free

- *or* add valgrind client hooks into fixed alloc api, so valgrind can monitor access directly of fixed-alloc memory

- maybe: also hook the GC alloc api's, for valgrind monitoring.

Since GC memory is zero'd by default, the main value of instrumenting the GC api would probably be bounds checking.  (requires padding on both ends of allocated blocks).

note: valgrind is supported on Linux x86, x64, ppc32, and ppc64.

Comment 1

[Edwin Smith]

Attachment: Suppressions file for mmgc conservative-mark noise

Comment 2

[Lars T Hansen]

As of version 3.5.0 (19 Aug 2009) valgrind works on MacOS X as well.  You do want to run with --dsymutil=yes to get line number information; why this is not the default is beyond me.

(The suppression file currently attached to the present bug does not seem to be useful for MacOS X runs; don't know if it's because the file is out of date or there's something else wonky.)

Comment 3

[Tommy Reilly]

only works on OS X 10.5 (or so configure tells me)

Comment 4

[Edwin Smith]

(In reply to comment #2)
> As of version 3.5.0 (19 Aug 2009) valgrind works on MacOS X as well.  You do
> want to run with --dsymutil=yes to get line number information; why this is not
> the default is beyond me.
> 
> (The suppression file currently attached to the present bug does not seem to be
> useful for MacOS X runs; don't know if it's because the file is out of date or
> there's something else wonky.)

possibly both.  I developed that suppressions file using a ubuntu x86 vmware vm, before valgrind supported macosx 10.5.

Comment 5

[Rick Reitmaier]

(In reply to comment #2)
> run with --dsymutil=yes to get line number information; why this is not
> the default is beyond me.
> 

FYI, the osx.project file also has 'strip debugging' information enabled; it should be disabled for all tamarin projects.

Comment 6

[Tommy Reilly]

Working on a patch to make valgrind know about our memory, the goal is to only have memory served from GC::Alloc and FixedMalloc be accessible and to lock everything else down as NOACCESS and using explicit unlocking when the GCHeap/GC/FixedMalloc needs to touch its private parts.

Comment 7

[Edwin Smith]

Attachment: Work around conservative stack scan

This patch makes tamarin valgrind-clean on simple tests, without any suppressions files.  Its based on an idea Julian Seward suggested:  in the conservative stack scan code, copy the stack value somewhere, mark somewhere as defined, then load that.  this suppresses errors within mmgc and leaves the stack value undefined, to continue catching mutator bugs.

Comment 8

[Edwin Smith]

A strawman plan would be to audit each place in MMgc we poison memory and instead (or in addition) add valgrind annotations.

Comment 9

[Julian Seward [:jseward]]

> Created attachment 457303 [details] [diff] [review]
> Work around conservative stack scan

Looks good.  Small thing:

+     static const void* launder = val;
+     VALGRIND_MAKE_MEM_DEFINED_IF_ADDRESSABLE(&launder, sizeof(launder));

By definition [&launder, +sizeof(launder)) is addressable, since you
just allocated it in the, er, data section, I guess.  So you might as
well just do VALGRIND_MAKE_MEM_DEFINED.

Also .. doesn't that make this thread-unsafe, even when not running on
Valgrind?  Shouldn't launder be stack allocated?  Ditto wrt 'scratch'
in previous chunk.

Comment 10

[Edwin Smith]

oh, (In reply to comment #9)
> > Created attachment 457303 [details] [diff] [review] [details]
> > Work around conservative stack scan
> 
> Looks good.  Small thing:
> 
> +     static const void* launder = val;
> +     VALGRIND_MAKE_MEM_DEFINED_IF_ADDRESSABLE(&launder, sizeof(launder));
> 
> By definition [&launder, +sizeof(launder)) is addressable, since you
> just allocated it in the, er, data section, I guess.  So you might as
> well just do VALGRIND_MAKE_MEM_DEFINED.

okay.  i was being conservative here.  In markItem(), we probably only want to
launder addresses if the mark work item was from stack (this is also marking
heap memory).

> Also .. doesn't that make this thread-unsafe, even when not running on
> Valgrind?  Shouldn't launder be stack allocated?  Ditto wrt 'scratch'
> in previous chunk.

oh, totally unsafe; i knew it would be, but using a local var would be fine, i have no idea why i didn't think of that.  i blame utter exhaustion.

Comment 11

[Julian Seward [:jseward]]

Attachment: revised patch

Revised as per comment 9.  Also:

* changes include file to "valgrind/memcheck.h"
  as per standard V install configuration

* nanojit/nanojit.h: don't redefine VALGRIND_DISCARD_TRANSLATIONS
  if it is already defined (not sure if this is correct, but required
  to get TR to compile)

Comment 12

[Edwin Smith]

Running on Snow Leopard with x86-64 build, I often see this:

UNKNOWN task message [id 3800, to mach_task_self(), reply 0xf03]

Comment 13

[Tommy Reilly]

(In reply to comment #12)
> Running on Snow Leopard with x86-64 build, I often see this:
> 
> UNKNOWN task message [id 3800, to mach_task_self(), reply 0xf03]

I see the same thing in 10.5 32 bit

Comment 14

[Tommy Reilly]

So I've spent a lot of time trying to get valgrind integrated and there's no easy way to do it.  

I started out thinking I would make it such that valgrind would generate an error anytime we touched any memory outside of what was returned from the MMgc alloc routines.  Sounds great in theory.   Touching any booktracking memory, block headers, page maps etc would generate an error.   

Doing this with client API requests wasn't too bad for GCHeap and FixedMalloc but probably too invasive to consider landing, definitely too invasive for the GC where you have object bits, page map, extensive block header manipulation code.   Doing it with client API requests requires literally hundreds of instances of placing VALGRIND_MAKE_MEM_NOACCESS/VALGRIND_MAKE_MEM_DEFINED calls around sections of code.  We have the freelists, the free blocks lists, the quick list, the object bits the block headers etc etc.    Also these functions all call each other in varying ways so you can't use a fine grained approach (ie if you unlock the block header in a function that does a bunch of interactions with it pieces of it become locked if you call a function that does a unlock/lock on a piece of the block header).  For the GCHeap locking/unlocking all the bookkeeping segments along with the heap lock acquisition/release worked well but GCHeap's bookkeeping memory is pretty simple (just the HeapBlocks array and the regions pages).

However you can't use suppressions because so many of our routines that touch this memory are inlined (ie FixedMalloc::Size) so we'd have to suppress all the methods that inline this in the mutator.   I'm now of the opinion that deep valgrind integration is too invasive to be worth it.   

Just defining MMGC_USE_SYSTEM_MEMORY makes little sense for tamarin (I tried it and it does work) but it might be more valuable in the player.  Either way GC memory is used to such a high percentage unless valgrind knows about GC memory we're missing out on a lot.

What might work is if you could do suppressions on ranges of code, ie make all memory inaccessible (except areas defined as allocations by MMgc using the client APIs) and then suppress all errors originating from MMgc.a.  Even this doesn't cover the inline cases though.

I noticed the firefox guys turn off jemalloc when they use valgrind because it isn't valgrindized either.

Comment 15

[Tommy Reilly]

Attachment: for posterity, where I left off, not working

Comment 16

[Edwin Smith]

For fixed-memory management, it's logical to map mmfx_new (etc) to malloc.  Can you extract a patch that does that?  (is there anything to patch, or do we already have the necessary feature switches?)

Comment 17

[Tommy Reilly]

--enable-use-system-malloc

Comment 18

[Nicholas Nethercote [inactive]]

(In reply to comment #13)
> > Running on Snow Leopard with x86-64 build, I often see this:
> > 
> > UNKNOWN task message [id 3800, to mach_task_self(), reply 0xf03]
> 
> I see the same thing in 10.5 32 bit

You can ignore this, it's just an internal Valgrind message.

Comment 19

[Tommy Reilly]

I don't think we want to blinding define everything MarkItem and the ZCT look at, I think we want to limit to the stack.   The reason is that it could be a bad bug if they look at uninitialized memory outside the stack.   For instance, one bug we've had is where a GCRoot doesn't initialize itself and it contains values from a previous allocation that prevent the GCRoot from ever being deleted (sometimes GCRoot deletion happens in a GCFinalizedObject dtor and if the root contains a pointer to that object then presto ... memory leak).

Comment 20

[Edwin Smith]

Does the ZCT scan look outside the stack?  I agree on both counts.   is the information available at the point when we're scanning (that the ptr is in the stack or not?)

Comment 21

[Tommy Reilly]

Yes, the ZCT scan also scans RCRootSegments, which I think the only current instance of is the WORD code interpreter stack.

What I've done is place the VALGRIND_MAKE_MEM_DEFINED calls in GC::DoMarkFromStack and ZCT::DoPinProgramStack for the entire stack region we're going to scan.   

After doing that I'm seeing some "Conditional jump or move depends on uninitialized value(s)" in MarkItem.    More specifically I see 3 of these errors running about a 1/4 into the ATS2 so I think these could be legit bugs.    Next I'll employ VALGRIND_CHECK_MEM_IS_DEFINED in MarkItem itself to see what allocations (ie which GCRoot) is looking uninitialized to valgrind.

Comment 22

[Edwin Smith]

You don't want to mark the stack itself as defined, because it will suppress user errors.  you only want to copy values off the stack, then mark the copies as defined (which is what the existing patch does).  Can we sneak a flag down to the mark loop to enable/disable the client request?

Comment 23

[Lars T Hansen]

(In reply to comment #21)
> Yes, the ZCT scan also scans RCRootSegments, which I think the only current
> instance of is the WORD code interpreter stack.

Also large alloca'd objects, I think, and all alloca'd objects on some systems (VMPI_alloca).

Comment 24

[Tommy Reilly]

I set up a repository here:  http://asteam.macromedia.com/hg/users/treilly/tr-valgrind

it runs most things okay but seeing some issues with acceptance tests on 10.5 with valgrind 3.6, will try on 10.6 as well.   

I didn't resolve on Uninitialized byte error in MarkItem, I worked around it with a suppression after digging for a couple days with no luck.   My .valgrindrc:

 --dsymutil=yes --suppressions=/Users/treilly/dev/asteam/tr-valgrind/MMgc/mmgc.supp --log-file=/tmp/valgrind-%p.txt --error-exitcode=1

Comment 25

[Edwin Smith]

Attachment: Second checkpoint from Tommy

Comment 26

[Felix S. Klock II [:pnkfelix, :fklock]]

Note: if valgrind has been installed into a non-standard location, then one will need to extend the set of include flags to g++ in order to tell it where to find "valgrind/memcheck.h"

I am getting by via manually hacking the generated Makefile and adding the "-I/usr/local/valgrind/include" to CXXFLAGS.  But we'll probably need an "--enable-valgrind" configuration option at some point, and when we add it, it should be able to handle valgrind being installed into non-standard paths.

Comment 27

[Felix S. Klock II [:pnkfelix, :fklock]]

(In reply to comment #24)
> I set up a repository here: 
> http://asteam.macromedia.com/hg/users/treilly/tr-valgrind
> 
> it runs most things okay but seeing some issues with acceptance tests on 10.5
> with valgrind 3.6, will try on 10.6 as well.   

How did things look for you on 10.6?  I gave it the tip of tr-valgrind a shot
on my Snow Leopard system, and saw a lot of acceptance test issues, but that
may be due to something about how I ran it.

Comment 28

[Tommy Reilly]

(In reply to comment #26)
> Note: if valgrind has been installed into a non-standard location, then one
> will need to extend the set of include flags to g++ in order to tell it where
> to find "valgrind/memcheck.h"

In my repo I just copied the headers into MMgc per Valgrind's docs recommendation, apparently these headers don't change very often by design and they are licensced liberally to make this workable.

Comment 29

[Felix S. Klock II [:pnkfelix, :fklock]]

(In reply to comment #28)
> (In reply to comment #26)
> > Note: if valgrind has been installed into a non-standard location, then one
> > will need to extend the set of include flags to g++ in order to tell it where
> > to find "valgrind/memcheck.h"
> 
> In my repo I just copied the headers into MMgc per Valgrind's docs
> recommendation, apparently these headers don't change very often by design and
> they are licensced liberally to make this workable.

But the MMgc.h header in your valgrind repo isn't pointing to that header, is it? 
See: http://asteam.macromedia.com/hg/users/treilly/tr-valgrind/annotate/b1faee3f8b08/MMgc/MMgc.h#l48

Comment 30

[Tommy Reilly]

sorry felix, I pushed my MMgc.h but not the move of the headers into a toplevel valgrind folder.  pull again

Comment 31

[Tommy Reilly]

status: I think I'm down to the SIGILL floating point illegal instruction errors and the JIT reading/writing below the stack pointer errors.  Doing another acceptance run now (takes hours).    If we indeed are at the point here's the plan:

- look at mmgc.supp, move what makes sense into client requests instead
- ifdef out valgrind completely (unless --enable-valgrind is specified) 
- rebase and get patch(es) ready for landing

Comment 32

[Tommy Reilly]

asteam.macromedia.com/hg/users/treilly/tr-valgrind is defunct, now using http://hg.mozilla.org/users/treilly_adobe.com/tr-valgrind/

which has been rebased to r 5116 (today)

Comment 33

[Julian Seward [:jseward]]

Possibly related: in the past few days, changes have been committed
in the Valgrind svn repo (svn://svn.valgrind.org/valgrind/trunk)
that make valgrind.h compilable using MSVC.  That means that in
principle your markup could be compiled into a Windows build, and
possibly, using a Valgrind/Wine stack, that could be Valgrinded.

Comment 34

[Tommy Reilly]

Attachment: add valgrind headers to tamarin

Comment 35

[Tommy Reilly]

Attachment: valgrind feature and client requests

Comment 36

[Tommy Reilly]

Attachment: valgrdind qe related changes

Comment 37

[Brent Baker]

Comment on attachment 476297 [details] [diff] [review]
valgrdind qe related changes

If you do not have RTARGS defined you cannot run with this patch. You can't attempt to access a key that does not exist, the check on line 136 has been bi-passed 
136: 
-            if os.environ.has_key(env)==False:
+            if os.environ.has_key(env)==False and env != 'RTARGS':


141:    val=os.environ[env]
You need to make sure that os.environ.has_key(env)

Comment 38

[Tommy Reilly]

Attachment: updates runsmokes.py to adequately deal with RTARGS

Comment 39

[Tommy Reilly]

Attachment: Refreshed to not have headers in patch

Comment 40

[Edwin Smith]

Comment on attachment 476295 [details] [diff] [review]
add valgrind headers to tamarin

Obviously no bugs, but since these don't have the mozilla tri-license, do they belong in other_licenses?  I think the guidelines are written down somewhere but I don't know where.

If they do belong in other-licenses, then the build scripts might need to be tweaked; ideally the includes in the source code don't pin down the include file locations.

as a data point, valgrind.h and memcheck.h don't exist in the moz codebase, maybe they just count on devs to always have it installed if building with valgrind enabled?

Comment 41

[Tommy Reilly]

(In reply to comment #40)

> Obviously no bugs, but since these don't have the mozilla tri-license, do they
> belong in other_licenses?  I think the guidelines are written down somewhere
> but I don't know where.

I'd hate for folks to have to edit the Makefile by hand to get this into the includes.   Maybe --enable-valgrind can append a -I$VALGRIND_HOME/include where VALGRIND_HOME defaults to /usr/local/valgrind?   Then folks could just set VALGRIND_HOME in their env (if necessary).

Comment 42

[Edwin Smith]

Agreed, it should be zero-hassle no matter what.  I'm not opposed to checking in the headers, just wanted to make sure they go in the right place.

Comment 43

[Tommy Reilly]

Attachment: Add valgrind headers to other-licenses

We still want the headers in there b/c we always include the headers and don't ifdef the client requests (looks nice).   If --enable-valgrind isn't on we define NVALGRIND which turns the client requests into no-ops.

Comment 44

[Tommy Reilly]

Attachment: Add client requests, valgrind feature and configure machinery

Updated to pick up headers from other-licenses

Comment 45

[Edwin Smith]

Comment on attachment 477931 [details] [diff] [review]
Add client requests, valgrind feature and configure machinery

R- only because I think this needs a SR? from Lars once the comments and nits are fixed, plus the dangling function calls guarding compiled-out valgrind client requests.  I couldn't find anything obviously wrong, and if it has stabilized when testing under valgrind, it should be ready to land.  I'm looking forward to using this daily!

Generally:
* new comments that are sentences should be capitalized and end with .
* Several places, new if statements are malformated -- need a space between if and (.  It looks like a lot of MMGC code is sloppy about this, but new code should conform to the desired style, old code notwithstanding.
* new client requests calls need spaces after ',' and around binary operations, for example:
   VALGRIND_MEMPOOL_FREE(lb,lb+1); should be
   VALGRIND_MEMPOOL_FREE(lb, lb + 1);

Somewhere obvious, there should be a design comment outlining what all the MMGC valgrind instrumentation is accomplishing, so that other MMGC devs know what is expected as the MMGC api's evolve.  In that comment, it would be helpful to include a one-line description of what each of the client requests does, maybe cut & paste from the valgrind docs, or put a link to the valgrind client request docs.

FixedAlloc-inlines.h: Can you reword this comment:

  // Note that we'd like to use "ask" size and not b->size but
  // the client to use the slop if it calls Size() but making this size and
  // expanding to b->size if Size is called isn't working, also the scanner
  // scans the full size regardless so we'd have to fix that first

It's rambling and alludes to the semantics of Size().  if "expanding to b->size if Size is called isn't working" then there should be a bug# and a fixme here pointing to the bug.

nit: you added braces around the call to memset, but i'm not sure why.  (by the way, should it be VMPI_memset?).

GC.cpp:
* is this call to wi.HasInteriorPtrs() guaranteed to optimize away if valgrind is disabled?

GCAlloc.cpp: in GCAlloc::FreeChunk, I was confused about calling VALGRIND_MEMPOOL_FREE twice, if (!m_bitsInPage) but only once otherwise.  I get it now (i hope) after reading the macro definition and sniffing around in the headers, so a comment would be good.  Something like "only mark b->bits as free if they were allocated separately".

GCHeap.cpp: Why call VALGRIND_MAKE_MEM_DEFINED() right before memset?  doing the memset should have the effect of making those bits defined.

GCObject.h: The comment for getCompositeSafe() needs capitalization and punctuation.

MMGC.h: a comment should explain why we disable MMGC_MEMORY_INFO if valgrind is enabled.  if its something we could fix in the future, include a FIXME and a bug number.

ZCT.cpp: 
* is the call to get->getStackTop() guaranteed to be optimized away if valgrind is disabled?

Comment 46

[Tommy Reilly]

Attachment: Updated to address Ed's comments

Comment 47

[Edwin Smith]

Comment on attachment 478014 [details] [diff] [review]
Updated to address Ed's comments

R+ because I don't need to review this again, still no bugs found.  But not all of my comments were addressed.  

The new comment in FixedAlloc-inlines.h still leaves a lot to be desired.
Here's my attempt (hint: commas go a long way):

  // We'd like to use the requested size, and not b->size, but
  // the client is allowed to use the extra bytes between the ask
  // size and b->size, after calling Size().  Using size here, and
  // resizing the block to b->size at the point the client calls Size()
  // isn't working.  See bug XXX, and watch Lars and Ed puke if a method
  // called Size() has the effect of calling realloc().
  // More importantly, the scanner scans the full b->size, so we'd have to
  // fix the scanner before worrying about the difference here.  (Bug##?)

I'd still like to see a comment in GCAlloc::FreeChunk.  

Numerous if statements are still misformatted.

GCHeap.cpp still has an apparently useless call to VALGRIND_MAKE_MEM_DEFINED right before memset.  Since memset also makes those pages defined, your comment doesn't explain why the mark-defined call is necessary.  doesn't calling memset make valgrind treat the pages as defined?

MMgc.h:

  // Valgrind integration is trickier with fresh memory scribbling and free memory
  // poisoning and its pointless since valgrind will uncover the same problems.

I don't see what MMGC_MEMORY_INFO and poisoning have to do with each other.  shrug,
i think i've flogged this pony enough -- but consider tossing in something for
us MMgc outsiders :-)

Comment 48

[Tommy Reilly]

Attachment: take 3, cleaned up nits and expanded comments

Comment 49

[Tommy Reilly]

Attachment: Take 4, really addressed all of Ed's comments this time

Comment 50

[Tommy Reilly]

(In reply to comment #47)

> The new comment in FixedAlloc-inlines.h still leaves a lot to be desired.

Fixed

> I'd still like to see a comment in GCAlloc::FreeChunk.  

Fixed
 
> Numerous if statements are still misformatted.

Fixed
 
> GCHeap.cpp still has an apparently useless call to VALGRIND_MAKE_MEM_DEFINED
> right before memset.  Since memset also makes those pages defined, your comment
> doesn't explain why the mark-defined call is necessary.  doesn't calling memset
> make valgrind treat the pages as defined?

Its not that the memory isn't DEFINED, its not even addressable, if the memory was used previously the memset looks to valgrind like a write to deleted memory.   

> I don't see what MMGC_MEMORY_INFO and poisoning have to do with each other. 
> shrug,

Historical bad naming

> i think i've flogged this pony enough -- but consider tossing in something for
> us MMgc outsiders :-)

No this is good, its not just me anymore so making everything clear actually has value ;-)

Comment 51

[Lars T Hansen]

Comment on attachment 478263 [details] [diff] [review]
Take 4, really addressed all of Ed's comments this time

To fix: In general I object strongly to the use of NVALGRIND, because it turns into a use of a double negative and that is just plain wrong.  (It was wrong from the day NDEBUG was invented.)  It is used just two places in all of these patches.  I suggest you convert those two #ifndef NVALGRIND to #ifdef MMGC_VALGRIND.  (If you need to leave the definition of NVALGRIND in then so be it, but we should not use it in MMgc code.)

To fix: Improper indentation in the #ifdef DEBUG nest in MMgc.h: comment should not be outdented where it is, and new #ifndef MMGC_VALGRIND should be indented.

To fix: Comment for getCompositeSafe in GCObject.h must be capitalized and punctuated.  Also need to be corrected: deleted objects cannot live on the stack, presumably what you mean is that pointers to deleted objects can live on the stack.

To fix: Comment in FixedAlloc-inlines.h should reference the discussion in bug #594756, you can probably cut down the comment in the .h file to a couple of lines and move the rest of the discussion, if any is needed, into that bug.

Another superreview won't be needed if all you do is fix the four items above.

Comment 52

[Tommy Reilly]

Addressed Lar's comments with some simple riders to fix various project files include paths and make msvc compilation work.

http://hg.mozilla.org/tamarin-redux/rev/55df5da12f64

Comment 53

[Tommy Reilly]

Some testing integration goodies:

http://hg.mozilla.org/tamarin-redux/rev/e684324824cf

Some basic docs:
http://hg.mozilla.org/tamarin-redux/rev/405b86fcd762

Comment 54

[Tommy Reilly]

no longer actively working on this, leaving open as it would be nice to get the nj hooks in and get a resolution on the hangs and integration of valgrind into testing is ongoing.

Comment 55

[Edwin Smith]

(In reply to comment #54)
> no longer actively working on this, leaving open as it would be nice to get the
> nj hooks in and get a resolution on the hangs and integration of valgrind into
> testing is ongoing.

by "nj hooks" do you mean bug 578673 and the other two dependent bugs?  If not, please elaborate or create more dependent bugs.

Comment 56

[Tommy Reilly]

(In reply to comment #55)
> by "nj hooks" do you mean bug 578673 and the other two dependent bugs? 

yes

Comment 57

[Brent Baker]

Tommy can this be closed out now? Testing of valgrind in tamarin is tracked via bug #600236

(In reply to comment #54)
> no longer actively working on this, leaving open as it would be nice to get the
> nj hooks in and get a resolution on the hangs and integration of valgrind into
> testing is ongoing.

What were the "hangs" that you were seeing? Is this still an issue?

Comment 58

[Tommy Reilly]

The hangs were on some tests I put in the testconfig.txt, see (In reply to comment #57)
> Tommy can this be closed out now? Testing of valgrind in tamarin is tracked via
> bug #600236

This is a tracker, do we close out trackers?

> What were the "hangs" that you were seeing? Is this still an issue?

Yeah, see the dependent bugs and the valgrind bits in testconfig.txt

Comment 59

[Sylvestre Ledru [:Sylvestre]]

No assignee, updating the status.

Comment 60

[Sylvestre Ledru [:Sylvestre]]

No assignee, updating the status.

Comment 61

[Sylvestre Ledru [:Sylvestre]]

No assignee, updating the status.

Comment 62

[Sylvestre Ledru [:Sylvestre]]

Tamarin is a dead project now. Mass WONTFIX.

Comment 63

[Sylvestre Ledru [:Sylvestre]]

Tamarin isn't maintained anymore. WONTFIX remaining bugs.